Crawler on update: Checksum remains unchanged even if HTML document has been modified

[stejacob]

Hi,

We are running a demo site on a Microsoft IIS web server and using the latest version of Norconex Crawler.

We've configured both the documentChecksummer and metadataChecksummer, but we’re noticing that the checksum value remains the same even after modifying the HTML file on the server.

We've tried using both the "Last Modified" field and the MD5 checksum on specific fields, but the document continues to be rejected because the checksum generated remains unchanged, even when the HTML document has been modified.

....
Line  8680: 14:12:28.256 [es-node2.deimscloud.mil.ca#3] INFO  REJECTED_UNMODIFIED - http://es-node2.deimscloud.mil.ca/about.html - MD5DocumentChecksummer - Checksum=4529f56e11c85023cd3b815ffd1c2b1e|

....
	Line 10840: 14:30:39.085 [es-node2.deimscloud.mil.ca#3] INFO  REJECTED_UNMODIFIED - http://es-node2.deimscloud.mil.ca/about.html - MD5DocumentChecksummer - Checksum=4529f56e11c85023cd3b815ffd1c2b1e|

Any insights or suggestions would be greatly appreciated. Thanks.

[essiembre]

Can you confirm something changed in the fields you use to create the MD5 checksum? A change anywhere else will not change the checksum.

If that is not the case, can you share your configuration?

[essiembre]

Also, the checksum is created AFTER the document is imported. That means you need to ensure the fields you use to create the checksum are still present in the document after it was imported.

[stejacob]

Morning Pascal,

Our configuration is:

....

<documentChecksummer class="MD5DocumentChecksummer"
                     combineFieldsAndContent="true"
                     keep="true"
                     toField="checksum">
  <fieldMatcher ignoreCase="false"
                ignoreDiacritic="false"
                method="CSV"
                partial="false"
                replaceAll="false">
        title,body_content,description</fieldMatcher>
</documentChecksummer>

.....

<metadataChecksummer class="com.norconex.collector.core.checksum.impl.GenericMetadataChecksummer" keep="true" targetField="metachecksum">
<sourceFieldsRegex>title|description</sourceFieldsRegex>
</metadataChecksummer>

Thanks Pascal.