Move all source responses to one subject

[dylanratcliffe]

I'm having issues with race conditions where a source will send a completion response, but the gateway will still be processing some of the items which will cause some of them to be dropped. The best way to solve this would be to have NATS send everything back to the requester on a single subject (items, responses and errors). NATS provides guarantees around ordering so this would solve our race condition

The only reason it's not like this in the first place was that I didn't know how to do oneOf in the protobuf message.

Implementing this would require:

• Create a new wrapper message type. Consider what else should be included in this message? Any metadata? Tracing?
• Change the docs to reflect the new naming convention for the new return subject
• Update sdp-go to use the new protobufs, to subscribe to the correct subjects and to process responses correctly
• Remove sleeps another other workarounds from sdp-go since we won't need them anymore
• Change discovery libraries to send on this new subject
• Update the permissions that are granted by api-server to ensure that sources and users can pub/sub to the new subjects
• Update all sources and gateway to use these new libraries

[dylanratcliffe]

I'm thinking about what the format of the response should be. The simplest thing to implement would be to have a message that is one of: Response, ItemRequestError or Item. But what would you call it? You can't really call it a response since it contains a response. Maybe it would be best that some of the things from Response were merged into here. For example it could be useful to know explicitly which responder an item came from, or maybe which request UUID it was related to without digging into the metadata

[dylanratcliffe]

If I were to move say the responder and the UUID up to the top level this might be good. However once we do that we then we might end up with something like this:

message ProgressUpdate {
  // The state of the responder
  ResponderState state = 2;

  // The timespan within which to expect the next update. (e.g. 10s) If no
  // further interim responses are received within this time the connection
  // can be considered stale and the requester may give up
  google.protobuf.Duration nextUpdateIn = 3;
}

Which isn't great because if you were to pass the whole message to a handler, it wouldn't contain enough information since it'd need the UUID and responder name

[dylanratcliffe]

Alternately we could do something like this, bring the UUID to the top but that's it:

message Response {
  // UUID if the item request that this response is in relation to (in binary
  // format)
  bytes itemRequestUUID = 1;

  oneof type {
    ProgressUpdate progressUpdate = 2;
    Item item = 3;
    ItemRequestError error = 4;
  }
}

This is a bit nicer since most of our handlers are going to be request-specific anyway so they can work out which request it's relevant to from the UUID then route the sub-message from there

[dylanratcliffe]

Another big advantage of this is that it looks like it might be possible to remove the subjects from the ItemRequest entirely and just rely on NATS response inboxes instead

[dylanratcliffe]

I've made some changes to SDP that look good. Working on sdp-go at the moment, next step is to add support for PublishRequest() and test that it definitely allows you to handle unsubscribing from the reply subject on your own terms