Include preview.png in zip download

[fulopkovacs]

⚠️ This PR is still a work in progress.

Notes

• Related issue: Include preview.png in zip download #14.
• I assumed that the preview.png file is the image that we fetch from https://cdn.polyhaven.com/asset_img/primary/${assetID}.png

[vercel]

@fulopkovacs is attempting to deploy a commit to the polyhaven Team on Vercel.

A member of the Team first needs to authorize it.

[fulopkovacs]

@gregzaal : this PR needs an intervention from you. Right now the CORS-headers of the preview images are not configured to allow requests from localhost:3002, so I see this error in Chrome (Firefox simply fails, doesn't print anything for me):

Access to fetch at 'https://cdn.polyhaven.com/asset_img/primary/modern_arm_chair_01.png' from origin 'http://localhost:3002' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

Could you fix the CORS-headers? Or is there a reason why https://dl.polyhaven.org/file/ph-assets/Models/png/1k/modern_arm_chair_01/modern_arm_chair_01_legs_nor_gl_1k.png can be fetched from localhost:3002, but https://cdn.polyhaven.com/asset_img/primary/modern_arm_chair_01.png not?

As always, sorry for the inconvenience, and take your time with the answer, it's not urgent. ☺️

[gregzaal]

I've added the CORS headers to all image files on cdn.polyhaven.com. There might be some stale caches around so let me know if there are any issues.

It would be better to use the thumbnail images instead of the preview, which would be https://cdn.polyhaven.com/asset_img/thumbs/${assetID}.png?format=png

The ?format=png is to avoid our image host automatically converting the file to webp, which some tools still don't support especially if the extension doesn't match.

[gregzaal]

I don't understand what this addition does - making the change here locally doesn't seem to do anything?

[fulopkovacs]

I get this error if I remove this condition:

The value of the data attribute of DownloadMap in this case is undefined, because it's supposed to be files[m][res] (see here), but files["preview"] has only these keys: "url", "path", "size", no resolutions.

[gregzaal]

No it's not. I believe it might be necessary to have the final bundle size known before downloading so this might be a roadblock.

[fulopkovacs]

Ok, so it's gonna be a tough one. For now, I hardcoded 0 for the size of the preview images, which is obviously a lie, but it's good for testing.

From what I see we have two problems here:

1. we need to get the file sizes of all the thumbnail images
2. we need to get that data into Firebase (if that's what you're still using)

For issue 1 I can use https://api.polyhaven.com/assets to get the list of all the assets, and then write a bash script that downloads them and uses exiftool to get their sizes. I can save this data to a csv/json.

As for issue 2, I don't have experience specifically with Firebase, but I believe we can write another script (that relies on firebase-tools (https://www.npmjs.com/package/firebase-tools)) to save the data in the db. This second script would have to run on your computer, because I have no access to Firebase. If you're using windows, then we might have to use Node.js instead of bash for this second script.

What do you think?

[fulopkovacs]

Okay, I realized that there's another, bit hacky way to solve this problem: https://github.com/Poly-Haven/polyhaven.com/pull/55/files#r1018417712. I don't recommend it, but we could use it if we don't have another option.

[fulopkovacs]

Okay, the CORS-headers seem to be fixed. The thumbnail images were included in the zip files in all my tests, so the only problem that we have now is that we don't know the size of the preview images (right now we lie about it, and say that it's 0), because they are not stored in the db. I proposed a solution for this issue here: #55 (comment).

[fulopkovacs]

I came up with this hacky way of getting the size of the preview images in the browser. I'd much rather fetch this data from the db, but I thought I should present this idea too.

Somewhat related, I noticed something strange: the size of the blob is the same as the size of the image if I download it manually (with wget), but the displayed size of the zip file is less than the size of the downloaded zip (8.2MB vs 7.78MB). A similar difference is present when I view the same file on the website (https://polyhaven.com/a/wood_cabinet_worn_long) too: 6.69MB vs 7.00MB. Maybe we don't add the size of a file to the sum that we display?

[gregzaal]

OK I think we can kill a few birds with one stone if we store the thumbnail images on our download server (where assets live, dl.polyhaven.org) as well as on the image server (cdn.polyhaven.com) and include those files in the database like the rest of them.

This way we solve Poly-Haven/Public-API#8, #57, and #14 (this issue) with known file size and no jank with bunny.net's webp conversion.

To actually achieve this requires some changes on our backend for future assets. and a script to retroactively upload existing asset thumbs, which is not something you can help with :( nor do I really know when I can make time for it.

Thanks for attempting this but it looks like it's not solvable in a nice way right now.

[fulopkovacs]

That's okay, I'll pick up something else. I don't think we'll ever use the last commit (that calculates the size of the image in the browser), so I removed it.

You can close this PR If you want, we can always reference it in the future.