Already-imported filter (for mass-upload gang)

Hey guys, I noticed a lot of you doing the same as me, and mass-importing certain queries from derpibooru.  
I created a filter on derpi that hides artists DNP’ed on pony.  
I also plan on hiding tags that have already been mass-copied over (so that the PhilomenaCopier program, that I suspect most of us are using, doesn’t spend unnecessary time checking if a picture has already been uploaded)  
This is the filter in question (hopefully you guys can access it)
 
The reason I’m posting about this on the forum is to ask you guys if you’ve dumped entire tags already. If yes, I could add them to the filter (I would check first), and save us all some time.
 
Of course, this approach wouldn’t be flawless, but I think it would be a good start to get most images that we care about onto here.

I can’t seem to access the filter.  
I went ahead and almost all of curtsibling’s art, since I felt it was some of the more at risk of getting taken off derpi.  
A few images didn’t seem to go through, however.  
Think I got all of Variant’s art, but might be some dupes.  
Got around half of the Flashimmer tag.  
got a bunch of the equstria’s best mother tag.

@Havoc  
Okay, added the two mostly-complete artist tags to the filter. Also, fixed the filter (it wasn’t public, it is now. It should work)

Thanks this should come in handy! Though you should really consider adding a lil reverse search query before uploading each image, it’s not perfect but it’ll be less work for the mods

@maxisb8  
Oh yeah, the program I use already checks that! I think sometimes a duplicate gets through, but usually not  

 
Edit: that said, it wouldn’t hurt to rummage around the source code and the philomena api to see if I can improve it

@sapphie  
I plan on downloading each image and applying a perceptual hash to them, philomena gives a lot of false positives with their reverse image search
 
We could also use the tags to try and see if the image was uploaded but wasn’t picked up by it
 
Also check out my importer :3  

@maxisb8  
Hmm. I was thinking of using the hash that’s present in the json files, since I downloaded the entirety of derpibooru’s images a few weeks ago.  
Once we have a substantial amount of images on ponybooru, I’d download the entirety of the json as well,  
Warning: nerd shit ahead  
and then use a data structure based on merkle trees to compute the symmetric difference between the two sets, then upload the images that aren’t on ponybooru yet. I even have the code already, and since it’s for an open source project I could definitely just recycle it.  
TL;DR: false positives aren’t an issue yet, because I’ll deal with them later

@maxisb8  
Lookin’ clean. I’m also very fond of the “idk just make a web page” approach for user interfaces

@sapphie  
I think futa is good to be put on that list since you’re uploading it and I’m almost done with the fat tag

@GooseGoose  
I was gonna put it on the list once I was done. I still have 12k images to go through (which should take around 36 hours)
 
What I guess I could do would be to add the tag while my program is running, since I do plan on finishing it.
 
Edit: nope. Adding the tag to the filter prevents further pages from being loaded, thus also limiting the uploads. So yeah, I’ll add it once I’m done uploading the tag.
 
 
Tell me when you’re done with that tag (generally would prefer avoiding adding a tag unless it’s already done with)
 
 
On a sidenote, once we’re done uploading things on a tag-by-tag basis, we could totally use this thread to coordinate ranged-based uploading (i.e. we cut the original image base into chunks of, say, 10k images, and keep track of which chunks have already been uploaded)

@sapphie  
That’s a great idea and I’ve got fat and obese done atm. Working on chubby. Eta is about 5 hours. I’ll also get my list of artists I’ve finished uploading after I get some sleep

Added to the filter!

Imported all of datte-before-dawn’s stuff. Adding him to the filter.  
Also importing all of shinycyan’s art and will add her to the filter when it’s done.
 
Edit: imported all of MKogwheel’s art.

I got all of Cosmonaut’s art, and I did everything I could get under the Analingus tag as well. I’m currently working on Creampies.

Cool, adding those!

@sapphie
 
@Snowquill  
I think we should focus on character tags after completing our respective uploads. It’d prob help to be able to eliminate some of the big character tags, like all the m6 and some bg pones. I know Trash Anon is uploading all the Celestia content and I’m starting Luna after I finish “Impossibly Large Belly”. Oh, and I finished chubby too

I’m currently manually going through my watched artists, and some small tags. Once I’m done with my current big tag I’ll set my VPS to a character tag.  
Calling dibs on Trixie.

Yeah, getting through character tags is going to be important. That will help us get through a lot of other tags as well. I’ll get Twilight next.

I’m Currently going through the Celestia tag

My creampies script died out and I’m having issues getting the syntax for how to have it pick up where it left off, so I’m going to find a really small tag till I can figure that out.

@Snowquill  
What I do in those cases is to just find the last picture I uploaded and find its id on derpi, then add id.gte:<id> to the query

@sapphie  
Thank you, you are a life saver. I’m not used to filtering through when something was posted, so I was just trying it on my own. Got to where I did it by year, but that’s even better since I know which post was the last one I posted from that.

@Snowquill  
Yeah, I did it by month once, was a huge pain.

I’m starting to have issues with the uploaded script. Seems to pop up an error when I try and start it, and closes itself. I’m about to go do something with my family, but if I can’t get this to start running again, I left off at creampie, id.gte:1533721 if anyone can see what’s going on.

Yeah, it works fine on my end (this is the creampie tag)  

 
Edit: went back to uploading my watched artists. Last image I uploaded was 1536236