HTTP binding in LwM2M 1.2 seems to be broken

[kFYatek]

I'm basing this issue on the approved version of LwM2M Transport TS 1.2 (OMA-TS-LightweightM2M_Transport-V1_2-20201110-A), section 7 "HTTP Transport Binding".

The problem

The "HTTP Transport Binding" sections describes a mapping from high-level LwM2M operations onto HTTP methods such as POST and GET.

Much like in the CoAP binding, little distinction is provided between requests sent from the client to the server (e.g. Register, Bootstrap-Request, Send) and those sent from the server to the client (e.g. Read, Write). In other words, it seems to mandate that both parties may act as either an HTTP Server or an HTTP Client, depending on the specific operation.

With CoAP, I'm not sure if this mode of operation was envisioned by the original authors of RFC 7252, but it's not explicitly forbidden, either. Moreover, since LwM2M has been one of the pioneering application protocols implemented on top of CoAP, this mode of operation has become an integral part of the LwM2M/CoAP ecosystem.

However, HTTP, that has been in common use for about 30 years, is widely understood as a protocol in which the request-response relationship is unidirectional; attempting to send requests in both directions over the same connection is likely incompatible with the governing RPCs, and most certainly incompatible with pretty much any existing implementation of the protocol.

The apparent message flow

According to the specification, the intended flow of communication consisting of a Register request from the LwM2M Client followed by a Read request from the LwM2M Server would be something like below. Note: > signifies data transmitted from the LwM2M Client to the LwM2M Server, and < signifies the opposite direction.

> GET /rd?ep=urn:dev:os:000000-example&lt=86400&lwm2m=1.2&b=H HTTP/1.1
> Host: example.com
> Content-Type: application/link-format
> Content-Length: 25
>
> </1/0>,</3/0>,</4/0>,</5>
< HTTP/1.1 201 Created
< Location: /rd/example
< Content-Length: 0
<
< GET /3/0/0 HTTP/1.1
< Host: 192.168.0.1
<
> HTTP/1.1 200 OK
> Content-Type: text/plain; charset=UTF-8
> Content-Length: 13
>
> Example Corp.

All of the above communication would happen over the same connection. As mentioned above:

• This is incompatible with majority, if not all, existing HTTP implementations - including:
  • libraries that could be used for implementing server and client software
  • infrastructure elements such as proxy servers, load balancers etc.
• This paradigm is untested, possibly leading to various corner cases if both parties attempt to send a request at the same time

While probably technically possible to implement, this seems to render most of the advantages of using HTTP moot. The main advantage of HTTP over CoAP+TCP or even MQTT is that HTTP is well understood, standardized and has a lot of readily available hardware and software produced around it.

Proposed solutions

The HTTP binding as described in TS 1.2 probably needs to be significantly amended, or perhaps even rewritten from scratch. I don't have any singular proposal for solving this issue. Below are a couple of alternatives that have been considered by me and my colleagues:

HTTP/2 Extension for Bidirectional Message Communication

There seems to be an Internet-Draft proposed by some people from Facebook that aims to standardize streams for reverse communication channel. The draft did not seem to receive much attention and has since expired - although it seems to have seen some adoption.

Advantages of this approach are minimal changes to the currently published specification. Disadvantages include reliance on HTTP/2 (whose adoption is itself limited, especially in the IoT industry) with obscure extensions, which may still limit the positive effects that stem from the fact that HTTP is a widely deployed protocol.

Solutions involving duplex connections

Another proposal that minimizes the amount of changes necessary in the TS would be to maintain not one, but two TCP streams for each client-server association; one used for client-to-server requests, and another for the return channel.

This allows usage of mostly standard HTTP/1.1 in both directions. However, specifying the way of maintaining association between the two streams would be a challenge. It would also most likely require two TLS handshakes for each client-server association, potentially wasting bandwidth.

WebSockets

The HTTP binding could be replaced with a solution based on the WebSocket protocol. The WebSocket stream could then carry CoAP messages as standardized in RFC 8323. That would make it possible for the WebSocket binding to reuse most of the behaviour specified for LwM2M over CoAP over TCP.

While the WebSocket protocol is distinct from HTTP, it is widely deployed and fairly well supported by modern web infrastructure. It might provide for similar benefits as true HTTP does, although not every advantage of HTTP is maintained that way, e.g. load balancing is most likely not easily possible.

Logical reversing of data flow

The CWMP (TR-069) protocol protocol solves a similar issue by mandating the client to send an empty HTTP POST message to logically reverse the flow of communication - the logical requests from the server are then sent in HTTP responses, and logical responses in further HTTP POST requests.

This technique could be straightforwardly adapted to support LwM2M queue mode operation - the client, after sending all outstanding server-bound messages (e.g. Register, Update, Send), would send such an empty POST message and let the server send outstanding client-bound messages (e.g. Read, Write).

However, that way the ability to use straightforward REST messages is lost, as the server cannot use HTTP methods such as POST and GET directly. These would need to either be mapped onto non-standard HTTP headers, or encapsulated within the content. I would propose using the CBOR-based encapsulation used for the MQTT binding for this purpose, just to reuse a format that has already been defined and to reduce size of the specification document.

Another disadvantage of this approach would be that non-queue mode operation cannot be modelled this way.

Long polling etc.

The problem with non-queue mode mentioned above could be addressed by using the informal "long polling" technique - after reversing the flow as described above, the client can wait an indefinite period of time for incoming logical requests (represented as HTTP responses).

The issue here would be that at any moment, a server-bound message may be generated. In that case the client would need to either close and re-establish the TCP/TLS stream, create another one without closing the existing one, or use HTTP/2 stream multiplexing.

Additional remarks

The TS also does not mention many aspects of HTTP communication that, while not essential for a proof-of-concept level of implementability, might be very important for production deployments - for example usage and lifetime of cookies used for communication. It may be worth noting that long-term cookie storage, in conjunction with TLS-based authentication, may allow relaxing requirements on performing the Register operation whenever a new TLS association is established.

Conclusion

Out of the proposals listed above, I would personally opt for standardizing LwM2M over WebSockets while removing support for "true" HTTP. However, I understand that this solution might not fit all use cases for which LwM2M over HTTP might be desirable.

The CWMP-style solution also seems fairly elegant to me. The HTTP binding is already quite limiting (no Obseve/Notify or Composite operations), so limiting it further to queue mode operation does not seem like a huge problem. This solution also seems the most friendly towards the intermediary HTTP infrastructure (proxies, load balancers etc.).

In any case, I strongly believe that this issue needs addressing to make HTTP binding actually useful and possible to implement.

[mkgillmore]

@PiotrRoszkowski Please have a look and let me know what you think