Is it safe to use {md5} as archive format for boorus?

[ClosedPort22]

I've downloaded a lot of stuff from various sites, and I'd like to check some boorus (primarily gelbooru_v01 and v02 instances) for missing images. To avoid downloading duplicate files and putting too much load on the servers, I'm considering hashing my local files and using the MD5 values (maybe with filesize, width, etc.) to manually build an archive database. But this may lead to false positives if the real MD5 of an image is different from the reported value. For example, suppose I have a low-quality version of an image, if someone uploaded a better version but somehow the MD5 didn't change, it will be skipped regardless. Is this a real concern or am I being paranoid? Thanks for any insights!

[mikf]

I don't really know, but I'd assume the reported md5 hash for identical files is identical across boorus. At least the hash should get updated when a file gets replaced with a better version, if replacing a file is even supported by that booru.

On rule34 for example, you can sometimes get a file with a different hash than what is reported:

$ md5sum rule34/rule34_7001507_42893394f024d0bdfd078dc0ffa8cc9b.jpg 
c482dd38c415c693144b156fa7e966b3  rule34/rule34_7001507_42893394f024d0bdfd078dc0ffa8cc9b.jpg

Or you get the "correct" version

$ md5sum rule34/rule34_7001507_42893394f024d0bdfd078dc0ffa8cc9b.jpg 
42893394f024d0bdfd078dc0ffa8cc9b  rule34/rule34_7001507_42893394f024d0bdfd078dc0ffa8cc9b.jpg

Also, you need to adjust archive-prefix if you want to use the same archive entries for multiple sites.

[ClosedPort22]

Thanks again for the quick response!

I'd assume the reported md5 hash for identical files is identical across boorus

It seems so, at least for the boorus that I'm interested in.

On rule34 for example, you can sometimes get a file with a different hash than what is reported

Hm, this sounds suspiciously like accidental file corruption, given that you can get the correct one. I was unable to reproduce the result; I always got the "correct" one. It could have something to do with Cloudflare if it happens more than once.

[Twi-Hard]

Different images can have the same hash. Like here
https://www.reddit.com/r/DataHoarder/comments/gokrmx/these_different_2_images_has_the_same_md5_hash/
I've heard of that happening on boorus too (with similar images).
Images posted to multiple boorus often have different hashes because they were processed differently upon upload. I've tested this with the philomena boorus which basically mirror each others content. To test it I downloaded the same search on multiple boorus with the same images on each site. I then checked the hashes of all of them and the amount of unique hashes was far greater than the amount of images in the search. I then processed all of the images (only works with pngs) the same way and it gave me the correct amount of hashes. Meaning the hashes didn't help much with not downloading duplicates.

[ClosedPort22]

Different images can have the same hash.

Yes, I'm aware that MD5 and SHA1 are no longer resistant to collisions. But to be fair, the chances of having accidental collisions are pretty small (and can be mitigated taking filesize, etc. into account), and I'm not too worried about people maliciously uploading images with forged MD5 hashes either.

Images posted to multiple boorus often have different hashes because they were processed differently upon upload.

That's interesting. I'm mostly worried about skipping over images that I don't have, so downloading duplicates is less of an issue for me.