Importer software thread (TODOs, requests, complaints, bug reporting, etc)

Sapphie
Rampant Squirrel - A ferocious friend
Autist -
Chatty Kirin - A user who has reached a combined 1000 forum posts or comments.
Lil Importer Shid - Hi, Im a lil shid.
Liberty Belle - Sings the song of the unchained

Moderator
Izzy Moonbow Enjoyer
Heya,
You probably know me as the crazy person who runs the derpibooru import bot. The bot's code can be found here

If you notice any strange behaviour (such as the bot not importing things it should be, or crashing), you can tell me about it here.

If you have any feature requests or suggestions, feel free to drop them in this thread.

Finally, the things I have planned and will eventually get to
- Fixing the SVG imports
- Improve the way I check for false duplicates. I'll have to look into how exactly philomena's reverse search works so that I can have a more confident result, since the site's reverse search functionality sometimes returns pictures that really shouldn't be matched. Maybe there won't be a way to check the images directly, but in that case I could probably check if artist and rating match up, and call that good enough.
- Add better logging
- Some videos don't seem to be importable using a link. My script wastes a lot of time trying to upload them. I should change that and have it detect when a video cannot be imported (I'd have to find some examples of that)
- Add twibooru compatibility (maybe; it's not an urgent issue, but twibooru uses the old API)
Marker
Liberty Belle - Sings the song of the unchained

This Sentence is False
Could you take a look at the links in the description of >>2732372 (NSFW)? It looks like there's an 's' appended to all URLs for image links using the thumbnail syntax.
Marker
Liberty Belle - Sings the song of the unchained

This Sentence is False
Heya! What does your tag_mapping config look like? You might want to add "comments locked down" to the list of removed tags.
Posted Report
Anonymous #8111
It looks like the import didn't happen today. Is that a bug, or was it put off for the imports from PonerPics?
Posted Report
Sapphie
Rampant Squirrel - A ferocious friend
Autist -
Chatty Kirin - A user who has reached a combined 1000 forum posts or comments.
Lil Importer Shid - Hi, Im a lil shid.
Liberty Belle - Sings the song of the unchained

Moderator
Izzy Moonbow Enjoyer
It's fixed.
Technical explanation:
I didn't change the import script to take command line argument changes I made to the copier into account.
Posted Report
Anonymous #850C
@Havoc
It's going again now. Checking where it stalled, it looks like it got stuck trying to import this WEBM

https://derpibooru.org/images/2548984
Sapphie
Rampant Squirrel - A ferocious friend
Autist -
Chatty Kirin - A user who has reached a combined 1000 forum posts or comments.
Lil Importer Shid - Hi, Im a lil shid.
Liberty Belle - Sings the song of the unchained

Moderator
Izzy Moonbow Enjoyer
@Anonymous #850C
Yeah, ponybooru is rather slow, so bigger files tend to time out, which leads my bot to try over and over again. I need to experiment a bit and see if it's possible to brute force my way around this my putting an absurdly high timeout window.
Anonymous #850C
Is it helpful to let you know when the importer is stalled on a larger file, or will it just skip the image after it tries long enough?

Also, how the hell is a 6000x6000 PNG over 88MB in size?: https://derpibooru.org/images/2551015
Marker
Liberty Belle - Sings the song of the unchained

This Sentence is False
@Anonymous #850C
Also, how the hell is a 6000x6000 PNG over 88MB in size?

Probably because it's saved in 16 bit/channel.
Opening the image in Photoshop and exporting it using 8 bit per channel reduced the size down to 20MB.
Sapphie
Rampant Squirrel - A ferocious friend
Autist -
Chatty Kirin - A user who has reached a combined 1000 forum posts or comments.
Lil Importer Shid - Hi, Im a lil shid.
Liberty Belle - Sings the song of the unchained

Moderator
Izzy Moonbow Enjoyer
@Anonymous #850C
Yeah, it automatically skips stuff after attempting it enough. I should probably up the timeout delay while reducing the max amount of retries. Maybe more images would get through, while the overall waiting time should stay the same.
Sapphie
Rampant Squirrel - A ferocious friend
Autist -
Chatty Kirin - A user who has reached a combined 1000 forum posts or comments.
Lil Importer Shid - Hi, Im a lil shid.
Liberty Belle - Sings the song of the unchained

Moderator
Izzy Moonbow Enjoyer
@Havoc
Nono, I know what's causing it go hang up. Big files don't process fast enough on pb, and as a result the importer loops (since it gets cancelled before it ever actually gets posted).
Interested in advertising on Ponybooru? Click here for information!
Graphic Design is my passion.

Ponybooru ain't free mate - help support us financially!

Syntax quick reference: *bold* _italic_ [spoiler]hide text[/spoiler] @code@ +underline+ -strike- ^sup^ ~sub~