AzureStorage: purge corrupted queue messages.

[jviau]

We should evaluate updating these two code locations to delete corrupted (fails deserialization) messages from the queue. It is not expected that deserialization failures is a transient issue and no amount of retries / time delay will fix these messages. Particularly because it is only framework types (and not user) being deserialized here.

Location 1:

durabletask/src/DurableTask.AzureStorage/Messaging/ControlQueue.cs

Lines 108 to 110 in b4ec695

	messageData = await this.messageManager.DeserializeQueueMessageAsync(
	queueMessage,
	this.storageQueue.Name);

Location 2:

durabletask/src/DurableTask.AzureStorage/Messaging/WorkItemQueue.cs

Lines 47 to 49 in b4ec695

	MessageData data = await this.messageManager.DeserializeQueueMessageAsync(
	queueMessage,
	this.storageQueue.Name);

[cgillum]

The one caveat to this policy is that we've seen cases where changes to Newtonsoft.Json settings can cause unintended deserialization failures. This can happen as part of a rollout of a new version of an app, whether due to changes made by the user (though hopefully we've rooted all those possibilities out) or changes made by the DTFx maintainers. Either way, giving time for users to roll back the change, e.g. 24 hours, before permanently deleting their data, might be prudent.

[jviau]

Yeah will need some design. It could be an opt-in setting? Or a callback? User gets the exception and gets to return true/false for purge?

Either way, the framework needs to take action here as it is not something users can self-mitigate (they will be fighting with the workers to dequeue and delete the message!)