How is content_hash computed for saved content?

[ndanner-wesleyancs]

I thought that the value of http_responses.content_hash is some hash of the content of the response to the request. However, I've come across two responses with the same content that nonetheless have different content_hash values.

Having done a crawl, I see the following:

sqlite3> select content_hash from http_responses where visit_id = <snip> and url = '<snip>' ;
26e0194c3609b91759298691d4838d45577b1667540263b2bb691ff2095b1461
de36e50194320a7d3ef1ace9bd34a875a8bd458b253c061979dd628e9bf49afd

I then extract the two results into files named after the hashes. Having done so:

$ diff 26e0194c3609b91759298691d4838d45577b1667540263b2bb691ff2095b1461.js de36e50194320a7d3ef1ace9bd34a875a8bd458b253c061979dd628e9bf49afd.js 
$ sha256sum 26e0194c3609b91759298691d4838d45577b1667540263b2bb691ff2095b1461.js 
3d82dbec1993140a53227dfec95fb4269d854f4b3da1bb0f2e7736538bd03091  26e0194c3609b91759298691d4838d45577b1667540263b2bb691ff2095b1461.js
$ sha256sum de36e50194320a7d3ef1ace9bd34a875a8bd458b253c061979dd628e9bf49afd.js 
3d82dbec1993140a53227dfec95fb4269d854f4b3da1bb0f2e7736538bd03091  de36e50194320a7d3ef1ace9bd34a875a8bd458b253c061979dd628e9bf49afd.js

So it must be that content_hash is being computed from something more than just the content.

Can anyone clarify this for me? Thanks!

[vringar]

Hey,
if they have the same content they should have the same hash in the database. So you've encountered a bug.
However, I don't have the time to look into this any further right now, so it would be great if you could file this as a bug and maybe try to collect some relevant context like where is hash calculated, is there any encoding or decoding happening afterwards?
Like does the content that gets stored handled any differently than the input to the hash function?

[ndanner-wesleyancs]

Might take me awhile before I get to this.