Purple Smart
Twilight is best pony
I’m aware that the reason so many pictures show “Thumbnail not yet generated” is because people are uploading pictures en masse. Is there any way to solve that issue, though?
Thumbnail not yet generated
Cloppy Hooves 
I imagine the only thing we can do is slow down with the uploads. I myself have been uploading my things one a at a time and only a handful each day. I was going to upload more today, but with how much strain the site is seemingly going through right now, I’ll wait.
The site is still in an early stage. It needs time to grow. It’s not the big, expensive server that Derpi has. In time, with enough support, this site can get there. We just need to spread more awareness of the shady shit Derpi staff does. We need to prove to people that this site isn’t just about Nazis and pedophiles. This is about abusive admins who shadow ban anyone who says anything that they don’t agree with.
The site is still in an early stage. It needs time to grow. It’s not the big, expensive server that Derpi has. In time, with enough support, this site can get there. We just need to spread more awareness of the shady shit Derpi staff does. We need to prove to people that this site isn’t just about Nazis and pedophiles. This is about abusive admins who shadow ban anyone who says anything that they don’t agree with.
Anonymous #5696
I don’t think that should be a concern. The people who are going to be dumb enough to believe that will take the hate mob’s lunatic ravings as their evidence, not actual thoughtful investigation. Anyone who does do real investigation and observation will quickly see that that isn’t the cast.
The marxists are losing their minds because their power from being a monopoly is being taken from them. Sanity, reason, and logic are literally anathema to them.
They’re going to hate you and say mean things about you no matter what you do. No matter what compromises or concessions you give them, it will only ever be a one-way street with them gaining ground and you losing it. There is no amount of appeasement that will satisfy a totalitarian tyrant.
Around 2.4 million images are going to be uploaded over the next few weeks or months. This is in addition to the normal “get a site up and running” issues. Also, remember that they have to import the image metadata, convert the large images to thumbnails and smaller versions of the image suitable for your display, and there’s plenty of stuff to keep the servers busy.
Frankly, I consider it a minor miracle that the sites are even usable in the first place.
@CloppyHooves
Frankly, I consider it a minor miracle that the sites are even usable in the first place.
@CloppyHooves
We need to prove to people that this site isn’t just about Nazis and pedophiles.
I don’t think that should be a concern. The people who are going to be dumb enough to believe that will take the hate mob’s lunatic ravings as their evidence, not actual thoughtful investigation. Anyone who does do real investigation and observation will quickly see that that isn’t the cast.
The marxists are losing their minds because their power from being a monopoly is being taken from them. Sanity, reason, and logic are literally anathema to them.
They’re going to hate you and say mean things about you no matter what you do. No matter what compromises or concessions you give them, it will only ever be a one-way street with them gaining ground and you losing it. There is no amount of appeasement that will satisfy a totalitarian tyrant.
Cloppy Hooves
Yeah, I guess you’re right. It’s probably for the best. The last thing we need is illiterate soyboy sheep joining the site. It just sucks. Derpibooru is making one grand a month on Patreon right now. It’s frustrating to watch people like them have so much money thrown at them.
I don’t think that should be a concern. The people who are going to be dumb enough to believe that will take the hate mob’s lunatic ravings as their evidence, not actual thoughtful investigation.
Yeah, I guess you’re right. It’s probably for the best. The last thing we need is illiterate soyboy sheep joining the site. It just sucks. Derpibooru is making one grand a month on Patreon right now. It’s frustrating to watch people like them have so much money thrown at them.
CurtainedWizard 
Devops
The Eponym
Yes, this issue will be solved forever when I’m allowed to resume work on the iron.
Right now this site is basically running in a toaster VM. We have a state of the art hypervisor server this service will be moving to. The iron this service is on right now is a primarily a file store, with a moderate Xeon v6 CPU. The new iron has a far far more powerful AMD Epyc CPU, and an ocean of RAM. If we need even more power later, we can upgrade the CPU to one that’s near twice as powerful, Rome 7542, and best of all the upgrade would have the metal down for just a few minutes.
Right now this site is basically running in a toaster VM. We have a state of the art hypervisor server this service will be moving to. The iron this service is on right now is a primarily a file store, with a moderate Xeon v6 CPU. The new iron has a far far more powerful AMD Epyc CPU, and an ocean of RAM. If we need even more power later, we can upgrade the CPU to one that’s near twice as powerful, Rome 7542, and best of all the upgrade would have the metal down for just a few minutes.
Purple Smart
That’s good to hear. Any ideas about how long it will take to move to that new server?
Twilight is best pony
Yes, this issue will be solved forever when I’m allowed to resume work on the iron.Right now this site is basically running in a toaster VM. We have a state of the art hypervisor server this service will be moving to. The iron this service is on right now is a primarily a file store, with a moderate Xeon v6 CPU. The new iron has a far far more powerful AMD Epyc CPU, and an ocean of RAM. If we need even more power later, we can upgrade the CPU to one that’s near twice as powerful, Rome 7542, and best of all the upgrade would have the metal down for just a few minutes.
That’s good to hear. Any ideas about how long it will take to move to that new server?
CurtainedWizard
Devops
The Eponym
@Purple Smart
I’m not really sure, because there is multiple factors involved, and two ways to interpret the question. There is only really one hold up, and the unexpected need to launch this service immediately halted my progress in resolving it.
How long will it take to:
Resolve that issue? I have no idea whatsoever.
Populate cloud with guests? A few hours.
Logistics: A few hours travel on a freeway I know well.
Installation: Since she’s replacing an older server, we have to take that one out of the rack first, and remove rails and cables and clean, all in a room thats a constant 95db of Nidec Ultraflo noise. Everyone loves those pics of datacenters, but those rooms are loud. Like really loud. Like so loud you could spend several grand on a server and instead of being all excited you spend literally weeks in dread making sure everything works perfect because you don’t want to go in that room more than you have to, and will do it right the first time at any cost. I have never once seen a YouTube vid that bears out the reality of the datacenter, it’s always some bespoke enterprise facility that’s clearly the CTO’s baby. Even this one where the guy is trying to make a point of their loudness is quite silver spoon. This is about the most accurate I can find. Imagine trying to debug a computer issue in that room.
@deathsia
By the time that becomes an issue we should be able to afford another colo. If not, we can rent cloud. But I prefer colo. Owned property is a lot more immutable and being the old fart I am, have seen too many services and datasets just go poof over the years.
I’m not really sure, because there is multiple factors involved, and two ways to interpret the question. There is only really one hold up, and the unexpected need to launch this service immediately halted my progress in resolving it.
How long will it take to:
Resolve that issue? I have no idea whatsoever.
Populate cloud with guests? A few hours.
Logistics: A few hours travel on a freeway I know well.
Installation: Since she’s replacing an older server, we have to take that one out of the rack first, and remove rails and cables and clean, all in a room thats a constant 95db of Nidec Ultraflo noise. Everyone loves those pics of datacenters, but those rooms are loud. Like really loud. Like so loud you could spend several grand on a server and instead of being all excited you spend literally weeks in dread making sure everything works perfect because you don’t want to go in that room more than you have to, and will do it right the first time at any cost. I have never once seen a YouTube vid that bears out the reality of the datacenter, it’s always some bespoke enterprise facility that’s clearly the CTO’s baby. Even this one where the guy is trying to make a point of their loudness is quite silver spoon. This is about the most accurate I can find. Imagine trying to debug a computer issue in that room.
@deathsia
By the time that becomes an issue we should be able to afford another colo. If not, we can rent cloud. But I prefer colo. Owned property is a lot more immutable and being the old fart I am, have seen too many services and datasets just go poof over the years.
CurtainedWizard
Devops
The Eponym
@Blossomforth
I won’t be alone, and I’ll bear the time just fine. I’m just treating it like a mission to space in regards to prep.
I won’t be alone, and I’ll bear the time just fine. I’m just treating it like a mission to space in regards to prep.
Husdur
As a suggestions to mods — maybe you should exclude such an images from displaying in lists.
On the main page and “trending” for sure, but I’d excluded them from “browsed” and “search” too.
I mean what’s the point of the list if it’s just “Thumbnails not yet generated” all over?
I think if you place notification above upload form, about why their image is not showing right away, users will understand.
On the main page and “trending” for sure, but I’d excluded them from “browsed” and “search” too.
I mean what’s the point of the list if it’s just “Thumbnails not yet generated” all over?
I think if you place notification above upload form, about why their image is not showing right away, users will understand.
hexado
BTW, some November 2015 — December 2019 derpibooru files have hash different from the one declared in JSON api/database (derpi forum post about this ), so you have to download them to have hashes. But you don’t even have to store them, you can redirect curl to sha512_sum even without storing to disk. If you have somewhere dump of derpibooru storage where you can run code, you can calclute hashsums locally.
You can store somewhere in logs information that file was taken from derpi, not optimized locally, if version otimized by old ffmpeg/gif code version has problems fixed in current code.
What if download optimized versions from derpibooru? When art is imported from derpibooru, it is useless to waste days of CPU clocks to optimize something already have been optimized.
if ((input file is huge PNG or animation) and (hash matches to actual derpibooru file data or to its source_hash)) {
take thumbs versions from derpibooru
if (input file hash matches to actual derpibooru file data) {
we dont even have to download full optimized version, we already have it here, the input file is already optimized;
} else { download optimized version from derpibooru }
}else {
process as usual
}
if ((input file is huge PNG or animation) and (hash matches to actual derpibooru file data or to its source_hash)) {
take thumbs versions from derpibooru
if (input file hash matches to actual derpibooru file data) {
we dont even have to download full optimized version, we already have it here, the input file is already optimized;
} else { download optimized version from derpibooru }
}else {
process as usual
}
BTW, some November 2015 — December 2019 derpibooru files have hash different from the one declared in JSON api/database (derpi forum post about this ), so you have to download them to have hashes. But you don’t even have to store them, you can redirect curl to sha512_sum even without storing to disk. If you have somewhere dump of derpibooru storage where you can run code, you can calclute hashsums locally.
You can store somewhere in logs information that file was taken from derpi, not optimized locally, if version otimized by old ffmpeg/gif code version has problems fixed in current code.
CurtainedWizard
Devops
The Eponym
I’ve rsync a complete copy of the originals to the server this site is actually meant to be on here at my house, and well as a database backup.
I will process the entire back catalog of missing thumbnails rather quickly.
I will process the entire back catalog of missing thumbnails rather quickly.
hexado
I looked into ./philomena-master/philomena-master/lib/philomena/processors/*.ex
Is it possible to use ‘large’ version for ‘medium’, ‘medium’ for ‘thumb’ and ‘thumb_tiny’ ? It doesn’t save encoding time, but saves some decoding. Also it possible to run ffmpeg in fastest mode for ‘large’ version and better reencode popular files later when the server become less busy. Also are WEBMs neccessery, is there devices which can play webm but have problems with mp4? I am looking at source code and see that webms are converted by vp8 codec, which has both quite bad quality and file size compared to mp4. Also ‘intensities’ for the ‘medium’ version is very likely the same as ‘full’.
What actually is the slowest part of the upload process, is it PNG or GIF or WEBMs optimization/thumbs or something else?
Is it possible to use ‘large’ version for ‘medium’, ‘medium’ for ‘thumb’ and ‘thumb_tiny’ ? It doesn’t save encoding time, but saves some decoding. Also it possible to run ffmpeg in fastest mode for ‘large’ version and better reencode popular files later when the server become less busy. Also are WEBMs neccessery, is there devices which can play webm but have problems with mp4? I am looking at source code and see that webms are converted by vp8 codec, which has both quite bad quality and file size compared to mp4. Also ‘intensities’ for the ‘medium’ version is very likely the same as ‘full’.
What actually is the slowest part of the upload process, is it PNG or GIF or WEBMs optimization/thumbs or something else?
Anonymous #C8F7
@hexado
Bad quality comes purely out of encoding settings, for any codec, even XVID. There’s always a compromise between speed of encoding and quality of the image. Being MP4 (aka h264 inside of MP4 container) changes nothing. VP8 and its brother VP9 require far less processing power while still retaining good results in both quality and the filesize than h264 does. I haven’t checked if the site actually outputs VP8 instead of VP9, but I can tell you this - switching to VP9 would improve the quality a bit but would slower down the speed of thumbnail generation.
You speak about “devices”, let me tell you about browsers. Alas, there’s a plethora of browser distros on Linux/Android that play Webm/VPx but ignore MP4/h264 because of its license restrictions, and vice versa - there are several browsers on Windows that kick out Webm and Ogg Vorbis/Opus support out of spite (or because MPEG consortium bribed them to do so).
With current FFMPEG’s optimizations, video processing takes actually less than or comparable to the “common” image formats, as it’s capable of taking advantage of the multicore/multiprocessor architecture. PNG, being just a container for Zlib-compressed data, has no advantage by design (aside the fact you can simultaneously compress 32 of them if you have enough cores/CPUs). JPEG, with its Huffman tables, while in theory can be accelerated, is still a complex beast at higher quality settings. GIF, being one of the oldest image formats out there with its LZW (iirc) encoding, is probably less optimized than anything else here.
Bad quality comes purely out of encoding settings, for any codec, even XVID. There’s always a compromise between speed of encoding and quality of the image. Being MP4 (aka h264 inside of MP4 container) changes nothing. VP8 and its brother VP9 require far less processing power while still retaining good results in both quality and the filesize than h264 does. I haven’t checked if the site actually outputs VP8 instead of VP9, but I can tell you this - switching to VP9 would improve the quality a bit but would slower down the speed of thumbnail generation.
You speak about “devices”, let me tell you about browsers. Alas, there’s a plethora of browser distros on Linux/Android that play Webm/VPx but ignore MP4/h264 because of its license restrictions, and vice versa - there are several browsers on Windows that kick out Webm and Ogg Vorbis/Opus support out of spite (or because MPEG consortium bribed them to do so).
With current FFMPEG’s optimizations, video processing takes actually less than or comparable to the “common” image formats, as it’s capable of taking advantage of the multicore/multiprocessor architecture. PNG, being just a container for Zlib-compressed data, has no advantage by design (aside the fact you can simultaneously compress 32 of them if you have enough cores/CPUs). JPEG, with its Huffman tables, while in theory can be accelerated, is still a complex beast at higher quality settings. GIF, being one of the oldest image formats out there with its LZW (iirc) encoding, is probably less optimized than anything else here.
hexado
@Anonymous #C8F7
VP8 gives pretty mediocre result compared to x264. Even best near-lossless quality settings sometimes produce corrupted shifted colors. ‘optimization’ process in philomena wich converts VP9 files to VP8 files gives “optimized” reencoded versions larger than input. Everything is wrong with VP8.
VP9 is pretty good codec, but 1) philomena uses VP8 (not 100% sure, just looking at ffmpeg command line in the source) 2) VP9 support is far from total in browsers.
I am sure there are browsers with every possible codec set combinations (including some in old TV which play only MPEG-2), but are they used to browse pony porn at pony boorus? If booru thumb/versions generation slows 2x time to support some browsers not used for actual web surfing on media sites, it is not best choice. I looked at https://caniuse.com/#search=mp4 to get conclusion thatevery most used browsers which can play vp8/wemb can play x264/mp4 too.
Also extra versions can be added later when ponybooru will stop struggle with massive export.
I can speculate that x264+vp9 is better than x264+vp8, but actually I don’t have web+video development experience.
Also we don’t know what actually slows down upload process, it can be something unxpected. Maybe videos are fine, but every SVG take 5 hours (just an example of possible unxpected bottlenecks). Or maybe optipng options are too extreme. Or one weird file forced specific version of ffmpeg to freeze for day. Or anything.
VP8 gives pretty mediocre result compared to x264. Even best near-lossless quality settings sometimes produce corrupted shifted colors. ‘optimization’ process in philomena wich converts VP9 files to VP8 files gives “optimized” reencoded versions larger than input. Everything is wrong with VP8.
VP9 is pretty good codec, but 1) philomena uses VP8 (not 100% sure, just looking at ffmpeg command line in the source) 2) VP9 support is far from total in browsers.
I am sure there are browsers with every possible codec set combinations (including some in old TV which play only MPEG-2), but are they used to browse pony porn at pony boorus? If booru thumb/versions generation slows 2x time to support some browsers not used for actual web surfing on media sites, it is not best choice. I looked at https://caniuse.com/#search=mp4 to get conclusion that
Also extra versions can be added later when ponybooru will stop struggle with massive export.
I can speculate that x264+vp9 is better than x264+vp8, but actually I don’t have web+video development experience.
Also we don’t know what actually slows down upload process, it can be something unxpected. Maybe videos are fine, but every SVG take 5 hours (just an example of possible unxpected bottlenecks). Or maybe optipng options are too extreme. Or one weird file forced specific version of ffmpeg to freeze for day. Or anything.
CurtainedWizard
Devops
The Eponym
The software issue on the server hypervisor is still vexing me so… fuck it! we’ll do it live. fucking thing sucks
Slapped in a spare SSD, rebooted, and just did the thing. Z deployed a sunbird, database is imported.
rsync updating for the last day….. wtf. apparently my local rsync lost some bits during some local transfers.
At this point, the fact that the thumbnailer wasn’t decoupled into a microservice I could hack up or replace has me feeling like… ah yes, this.
In other news there is an upgrade on eBay we could really use that generally cost about six hundred but someone is selling for two fifty. And I’m broke. Guess we will need a fundraiser. Oh hey! Fridge magnets. Oh, apparently it sold it the time it took for me to type up this post. Well that’s disappointing. Oh well. Swag sale soon-ish, for those of you who believe in us and want a way to contribute.
After this final rsync is complete, the great thumbnailing catchup begins at Warp 9.9975. tantrum will be running it with 40/0 threads versus skystar’s 4/2, for images/video respectively. Also pure SSDs, versus a slow RAID5 of spinners. By this time tomorrow the back catalog should be wiped out.
This post brought to you by the Reddit Spacing To Piss Off That One Guy On 4chan™ gang.
Slapped in a spare SSD, rebooted, and just did the thing. Z deployed a sunbird, database is imported.
rsync updating for the last day….. wtf. apparently my local rsync lost some bits during some local transfers.
At this point, the fact that the thumbnailer wasn’t decoupled into a microservice I could hack up or replace has me feeling like… ah yes, this.
In other news there is an upgrade on eBay we could really use that generally cost about six hundred but someone is selling for two fifty. And I’m broke. Guess we will need a fundraiser. Oh hey! Fridge magnets. Oh, apparently it sold it the time it took for me to type up this post. Well that’s disappointing. Oh well. Swag sale soon-ish, for those of you who believe in us and want a way to contribute.
After this final rsync is complete, the great thumbnailing catchup begins at Warp 9.9975. tantrum will be running it with 40/0 threads versus skystar’s 4/2, for images/video respectively. Also pure SSDs, versus a slow RAID5 of spinners. By this time tomorrow the back catalog should be wiped out.
This post brought to you by the Reddit Spacing To Piss Off That One Guy On 4chan™ gang.
CptConundrum
@CurtainedWizard
>This post brought to you by the Reddit Spacing To Piss Off That One Guy On 4chan™ gang.
REEEEEEEEEEEEEEE
>This post brought to you by the Reddit Spacing To Piss Off That One Guy On 4chan™ gang.
REEEEEEEEEEEEEEE
This topic has been locked to new posts from non-moderators.
Locked
Lock reason: Resolved.