Importer software thread (TODOs, requests, complaints, bug reporting, etc)

Anonymous #A94A
I’ve noticed some gaps from the last import.
 
Between here and here there were five more images that got omitted in the import, none of them are from DNP artists, of overly large file size, or any other obvious qualities that should have let them be skipped over. Another three were lost between here and here, then a few images skipped ever other image after that. I could keep chasing this, but it looks like at least for today’s import several images were being skipped randomly. The image numbers aren’t showing any gaps, so I know they aren’t hidden or haven’t been merged from an early upload, I already checked some of the artist’s of the overlooked pieces for that possibility.
Sapphie
Rampant Bear - A courageous companion
Autist -
Chatty Kirin - A user who has reached a combined 1000 forum posts or comments.
Lil Importer Shid - Hi, Im a lil shid.
Liberty Belle - Sings the song of the unchained

Senior Moderator
Unavailable
@Anonymous #A94A  
Looking into it now, this is the output I got.  
Uploading image 673/715 (2647347)  
Error uploading image (400)  
{"errors":{"image":["can't be blank"],"image_aspect_ratio":["can't be blank"],"image_duration":["can't be blank"],"image_format":["can't be blank"],"image_height":["can't be blank"],"image_is_animated":["can't be blank"],"image_mime_type":["can't be blank"],"image_orig_sha512_hash":["can't be blank"],"image_sha512_hash":["can't be blank"],"image_size":["can't be blank"],"image_width":["can't be blank"],"uploaded_image":["can't be blank"]}}  
{'image': {'description': 'Image imported from [derpibooru](https://derpibooru.org/2647347\n(No) description on original)', 'source_url': 'https://derpibooru.org/2647347', 'tag_input': 'caption, dog, edit, edited screencap, equestria girls, image macro, safe, screencap, spike, text, twilight sparkle, sunset shimmer, spike the dog, equestria girls (movie), derpibooru import'}, 'url': 'https://derpicdn.net/img/view/2021/7/1/2647347__safe_edit_edited+screencap_screencap_spike_sunset+shimmer_twilight+sparkle_dog_equestria+girls_equestria+girls+28movie29_caption_image+macro_spike+the+.jpg'}  
Looks like philomena bullshittery to me, given the “image can’t be blank” stuff. I’m gonna try to upload those five pics manually, and I’ll keep y’all posted.
Sapphie
Rampant Bear - A courageous companion
Autist -
Chatty Kirin - A user who has reached a combined 1000 forum posts or comments.
Lil Importer Shid - Hi, Im a lil shid.
Liberty Belle - Sings the song of the unchained

Senior Moderator
Unavailable
Yeah what the fuck. I just re-ran the importer on that range of images, and it had no issues uploading everything. So either pb’s servers were momentarily fucky wucky, or philomena shat itself. I’m thinking I could try to do a second round of imports every day (which won’t cause any dupes since it’ll just give “hash taken” errors and continue), about an hour after the first round starts.  
Edit: Oh wait, I could also do it the smart way and check what kind of error it gives me. I’ll probably get to that in the final iteration of the importer software, since i’d need to rewrite it from ground up I think
Marker
Boot badge - It's Bootiful
Fried Chicken - Attended an April Fools event
Liberty Belle - Sings the song of the unchained

Misanthrope
Hello Sapph.  
Just made some changes to my importer script, it will now add the tag ‘ponybooru import’ to images uploaded from here to Derpi by default. You may want to add it to your filter so those images won’t get picked up by your daily imports.
Sapphie
Rampant Bear - A courageous companion
Autist -
Chatty Kirin - A user who has reached a combined 1000 forum posts or comments.
Lil Importer Shid - Hi, Im a lil shid.
Liberty Belle - Sings the song of the unchained

Senior Moderator
Unavailable
Fucking pain.  
I have to find a way (and the time to do so) to translate derpi’s markdown to philomena’s (admittedly retarded) markdown.  
Thankfully there’s probably a whooooole bunch of markdown parsers out there, but writing the translation code sounds like pain.  
I’m still pissed at philo maintainers for pushing such a useless fucking change while philomena isn’t even at feature parity with BoR. Sure, is the markdown they use now better than the homebrew garbage they had before? Yeah why not. Was it a change that was necessary in any way, shape, or form? Fuck no.
Sapphie
Rampant Bear - A courageous companion
Autist -
Chatty Kirin - A user who has reached a combined 1000 forum posts or comments.
Lil Importer Shid - Hi, Im a lil shid.
Liberty Belle - Sings the song of the unchained

Senior Moderator
Unavailable
Okay, got a look at the logs. For some reason pb is shitting itself when the script tries to upload an image. I get an error 400 for any image I try to upload. Will dig into it this afternoon.
Anonymous #3A69
Speaking of the importer, for images that aren’t a good fit for Derpi, on which booru should I upload them so the other boorus can copy them fastest? Does this booru also import from Twibooru and Ponerpics or do you only ingest from Derpi?
 
@Sapphie  
It wasn’t “Homebrew garbage” Markdown they had before. It was a half-implemented parser for Textile. I’m still waiting for some crazy dude to write a LaTeX parser for their booru.
Sapphie
Rampant Bear - A courageous companion
Autist -
Chatty Kirin - A user who has reached a combined 1000 forum posts or comments.
Lil Importer Shid - Hi, Im a lil shid.
Liberty Belle - Sings the song of the unchained

Senior Moderator
Unavailable
@Anonymous #3A69  
Pretty much this one.  
Iirc ponerpics imports from pb as well as the other booru, but I remember seeing something in the altbooru thread about twibooru planning on doing the same once he figures out how to manage the dupes  
And nope, pb only grabs from derpi
Anonymous #3A69
@Sapphie  
Twi has a ponerpics import tag, so I think they’ve figured the mirror out. That would make the uploading decision tree go as follows:  
  1. If it’s safe to upload to Derpi, upload to Derpi  
  2. If it’s DNP for both Derpi and here, take your pick as to whether you like PP or Twi more  
  3. Otherwise, upload here
Anonymous #3A69
@Anonymous #3A69  
Both an image I uploaded here and an entirely separate image I uploaded to Ponerpics were not successfully mirrored to Twibooru. Has anyone else had images that Twibooru didn’t successfully import? I even checked for both using the everything filter, so it wasn’t a case of getting caught by an errant tag.
 
(It turns out that Ponerpics did successfully import the image I uploaded here. I am just a chromosome-hoarder and tagged it with something that was removed by my filter. Switching to Everything to test showed the successful import in all its disreputable glory.)
Anonymous #3A69
@Anonymous #3DAD  
Thanks for the info. Are there other automatic imports I should know about (or processes I think are automated but are, in fact, manual)?  
If I remember, I’ll make a graph this afternoon showing the flow of imports.
Syntax quick reference: **bold** *italic* ||hide text|| `code` __underline__ ~~strike~~ ^sup^ %sub%

Detailed syntax guide