Importer software thread (TODOs, requests, complaints, bug reporting, etc)

Sapphie
Rampant Bear - A courageous companion
Autist -
Chatty Kirin - A user who has reached a combined 1000 forum posts or comments.
Lil Importer Shid - Hi, Im a lil shid.
Liberty Belle - Sings the song of the unchained

Senior Moderator
Unavailable
Heya,  
You probably know me as the crazy person who runs the derpibooru import bot. The bot’s code can be found here
 
If you notice any strange behaviour (such as the bot not importing things it should be, or crashing), you can tell me about it here.
 
If you have any feature requests or suggestions, feel free to drop them in this thread.
 
Finally, the things I have planned and will eventually get to  
  • Fixing the SVG imports  
  • Improve the way I check for false duplicates. I’ll have to look into how exactly philomena’s reverse search works so that I can have a more confident result, since the site’s reverse search functionality sometimes returns pictures that really shouldn’t be matched. Maybe there won’t be a way to check the images directly, but in that case I could probably check if artist and rating match up, and call that good enough.  
  • Add better logging  
  • Some videos don’t seem to be importable using a link. My script wastes a lot of time trying to upload them. I should change that and have it detect when a video cannot be imported (I’d have to find some examples of that)  
  • Add twibooru compatibility (maybe; it’s not an urgent issue, but twibooru uses the old API)
Marker
Boot badge - It's Bootiful
Fried Chicken - Attended an April Fools event
Liberty Belle - Sings the song of the unchained

Misanthrope
Could you take a look at the links in the description of >>2732372 (NSFW)? It looks like there’s an ‘s’ appended to all URLs for image links using the thumbnail syntax.
Marker
Boot badge - It's Bootiful
Fried Chicken - Attended an April Fools event
Liberty Belle - Sings the song of the unchained

Misanthrope
Heya! What does your tag_mapping config look like? You might want to add “comments locked down” to the list of removed tags.
Anonymous #8111
It looks like the import didn’t happen today. Is that a bug, or was it put off for the imports from PonerPics?
Sapphie
Rampant Bear - A courageous companion
Autist -
Chatty Kirin - A user who has reached a combined 1000 forum posts or comments.
Lil Importer Shid - Hi, Im a lil shid.
Liberty Belle - Sings the song of the unchained

Senior Moderator
Unavailable
It’s fixed.  
Technical explanation:  
I didn’t change the import script to take command line argument changes I made to the copier into account.
Sapphie
Rampant Bear - A courageous companion
Autist -
Chatty Kirin - A user who has reached a combined 1000 forum posts or comments.
Lil Importer Shid - Hi, Im a lil shid.
Liberty Belle - Sings the song of the unchained

Senior Moderator
Unavailable
@Anonymous #850C  
Yeah, ponybooru is rather slow, so bigger files tend to time out, which leads my bot to try over and over again. I need to experiment a bit and see if it’s possible to brute force my way around this my putting an absurdly high timeout window.
Marker
Boot badge - It's Bootiful
Fried Chicken - Attended an April Fools event
Liberty Belle - Sings the song of the unchained

Misanthrope
Also, how the hell is a 6000x6000 PNG over 88MB in size?
 
Probably because it’s saved in 16 bit/channel.  
Opening the image in Photoshop and exporting it using 8 bit per channel reduced the size down to 20MB.
Sapphie
Rampant Bear - A courageous companion
Autist -
Chatty Kirin - A user who has reached a combined 1000 forum posts or comments.
Lil Importer Shid - Hi, Im a lil shid.
Liberty Belle - Sings the song of the unchained

Senior Moderator
Unavailable
@Anonymous #850C  
Yeah, it automatically skips stuff after attempting it enough. I should probably up the timeout delay while reducing the max amount of retries. Maybe more images would get through, while the overall waiting time should stay the same.
Sapphie
Rampant Bear - A courageous companion
Autist -
Chatty Kirin - A user who has reached a combined 1000 forum posts or comments.
Lil Importer Shid - Hi, Im a lil shid.
Liberty Belle - Sings the song of the unchained

Senior Moderator
Unavailable
@Havoc  
Nono, I know what’s causing it go hang up. Big files don’t process fast enough on pb, and as a result the importer loops (since it gets cancelled before it ever actually gets posted).
Anonymous #57A0
About the new Alt Links that have been included in the imports recently. Hovering my cursor over them shows them pointing to an image on ponerpics, but when I click on them I just get sent to the front page of ponybooru.
Sapphie
Rampant Bear - A courageous companion
Autist -
Chatty Kirin - A user who has reached a combined 1000 forum posts or comments.
Lil Importer Shid - Hi, Im a lil shid.
Liberty Belle - Sings the song of the unchained

Senior Moderator
Unavailable
@Anonymous #57A0  
Yeah, that makes sense. Ponerpics is supposed to have id matching with derpi, but that’s not in place for the recent (december till now) uploads. In the future those links will lead to mirrored posts on poner (at least if ponerpics’ admins stay true to their word)
 
I also changed the relative link translation. I need to change it back since it reduces usability (or maybe I’ll add an alt link instead, though that wouldn’t be great either)
Syntax quick reference: **bold** *italic* ||hide text|| `code` __underline__ ~~strike~~ ^sup^ %sub%

Detailed syntax guide