Flag to coerce missing Maybe fields in records to Nothing in ports

[rehno-lindeque]

I'd like to suggest that some completely optional flags like

elm make --coerce-missing-fields-to-nothing --coerce-undefined-fields-to-nothing

could do a lot to help people with existing code bases integrate with Elm in a robust manner.

For context, the suggestion I'm making is that these flags should probably not be used in testing / development / qa environments, but that they may be very useful for compiling embedded widgets to use in a production environment where Elm's promise of "no runtime errors" really counts.

Rationalle

Currently, when there's a field missing from a record being passed into an Elm port, the result is a run-time error. This is slightly confusing for many JavaScript developers as they routinely abuse the undefined value that JavaScript returns for record.missingField by treating it as though it is just a null value.

Now, I actually believe that Elm implements the "correct" behavior by informing you that you have made a mistake and that an undefined / missing field is a programmer error. So I've been working with the current behavior for quite some time, attempting to rigorously pass correct schemas to our widgets. Unfortunately, despite my attempts to be rigorous we've had a couple of embarrassing instances where visitors to our site have run into error messages like these:

While I appreciate that this is certainly something that needs to be fixed, I would much prefer to run into it in testing / qa at some later date. 99% of the time the error also really has no bearing on the functionality of the end-product and no harm would come from simply treating the missing field as null (if the field is already specified as a Maybe and receiving null values is expected behavior).

It is also very difficult to enforce those constraints on 3rd party JavaScript libraries where leaving fields off of the result is considered to be an every-day practice. It is sometimes very difficult to ascertain the precise return type of a function, I've found that even formal API documentation doesn't always specify which fields in a return type are optional.

To be clear, I think that this is an inherent problem with JavaScript's lax semantics, not with Elm. However, although the internals of an Elm library is beautifully robust there's no way to statically enforce type-safety of data passing through ports, with the end result that the interface with JavaScript becomes rather fragile. The only other way to ensure the that the JS <-> Elm interface is used in a safe way is by doing a great deal of extremely complicated end-to-end integration testing (which puts a very large maintenance burden on developers, even after the tests have all been written).

A lot of people are also running NodeJS/Ruby/etc servers behind their sites where these sorts of inconsistencies permeates their code. It's pretty irritating when those sorts problems manifest themselves in user-facing Elm widgets instead of the original code that was the perpetrator of all this awfulness :)

[Apanatshka]

Can you work around this by doing JSON parsing in Elm where you turn missing fields into Nothings?

[rehno-lindeque]

I could if ports are treated as JSON de/serializers... (I'd be a little bit sad about the added boilerplate, but not the end of the world)

[Apanatshka]

@rtfeldman gave a little talk about how he does a lot of JSON (de)serialisation in Elm for NoRedInk. But I think that's mostly because of server communication rather than direct communication with JS. I'm not sure if you can easily send arbitrary JSON from JS to Elm with ports (I haven't worked with that in a long time) and do (de)serialisation in Elm. You may need to create a JSON string in JS and parse that in Elm, which might incur some overhead, and isn't exactly pretty.

[mgold]

To be clear, this would only apply to ports of a Maybe a type, right?

I'd almost prefer to roll these into a --production flag, but that's splitting hairs.

As for a workaround, if you control the JS interface, maybe take in an object and enforce that it has all the rights fields before passing it to Elm? This would probably allow you to abstract the Elm initialization code away from the JS clients, perhaps down to something like myWidget(rootElement, options).

[rehno-lindeque]

You may need to create a JSON string in JS and parse that in Elm, which might incur some overhead, and isn't exactly pretty.

True, I've thought a little bit about what that might have looked like because it would be pretty convenient to supply defaults for other types of fields too (which is easy to do with the json stuff). However, I personally have a feeling that ports might eventually want to allow anything that supports structured clone for pragmatic reasons that aren't apparent now.

We were also thinking of doing communication directly to Elm via Ajax, however it wasn't quite ready for us at the time and I think there's a promising use-case for more-or-less stateless Elm widgets to be embedded inside other UI frameworks. (We currently do this in Angular and it works pretty well aside from the occasional schema clash and a bit of boilerplate).

[rehno-lindeque]

To be clear, this would only apply to ports of a Maybe a type, right?

That's exactly it, yes.

I'd almost prefer to roll these into a --production flag, but that's splitting hairs.

I was thinking that maybe you might even want it to be a bit obscure to avoid abuse, but I'm not sure.

As for a workaround, if you control the JS interface, maybe take in an object and enforce that it has all the rights fields before passing it to Elm?

This is pretty much what we do right now when something like this pops up, it would be easier to do if Elm reflected the available ports and their types so that you can do it automatically. Some of our objects have quite a lot of fields - it's not always all that convenient to enumerate them.

[mgold]

it would be easier to do if Elm reflected the available ports and their types

This wouldn't require a compiler setting that can affect the semantics of the code, so I think this approach may be more viable.

[rehno-lindeque]

it would be easier to do if Elm reflected the available ports and their types

This wouldn't require a compiler setting that can affect the semantics of the code, so I think this approach may be more viable.

For reference, related to Reflect available ports in JavaScript?, though this would require deeper reflection as the fields are typically in records. Not sure how easy/nice it would be to expose all that type information to JS.

I imagined something similar to this might be implemented for JSON if we work on automatically deriving JSON parsers in the future (perhaps structured clone is not entirely different from JSON parsing - I could imagine a similar applicative-style interface).

[rtfeldman]

@Apanatshka As it happens, we actually do read JS values through a port and decoder! 😸

The way we get the initial Model is by the server rendering a big JSON blob in the raw HTML that gets sent to the client (we still render bespoke HTML on every page request...and it's a very long road before that can stop being the case for us), and then a tiny bit of JS code grabs that JSON of the DOM and sends to Elm.

Here's the Elm code:

port sourceJson : Value


main : Signal Html
main =
    case Decode.decodeValue ModelDecoder.model sourceJson of
        Ok model ->
            userInputs
                |> Signal.foldp Form.update model
                |> Signal.map (view actions.address)

        Err error ->
            Signal.constant viewInvalidJsonError

[rtfeldman]

As far as flags go, I don't have strong feelings about ports and coercing, but I actually have meant to bring up the fact that it's pretty bad to spit out error messages like this directly into the DOM.

It's super unprofessional for an end user to ever see what's depicted that screenshot, but currently that's entirely possible if something goes wrong...and there's no clean way to prevent it from happening. What you want end users to see is something like "Sorry! Something went wrong. Don't worry, though - our support team is on the case! In the meantime, you can..." etc.

That said, this could be accomplished without a flag. For example, the Elm runtime could emit a elm.porterror event (which exposes some preventDefault analog that lets you say "hey Elm, this is production so we'll take it from here; don't actually print the error message to the DOM"), so you could have some JS boilerplate that listens for that event and handles things in a nicer way.

@evancz I forgot to mention this to you, but if you're open to the idea I'd be happy to open an issue for it in the elm-html repo.

[mgold]

@rtfeldman Or, the less technical error messages could be part of a --production flag. Just sayin'...

[rtfeldman]

Personally I really appreciate a simple CLI, so in general I'd rather avoid adding compiler flags unless there's really no good alternative. In this case, though, there seems to be a pretty straightforward alternative!

[evancz]

I am trying to clean up the issues on these repos. My goal is for issues on repos like elm-compiler and elm-repl are about specific bugs. It is not working to manage proposals and stuff that requires changes to the language here, so I am devising a way of keeping track of all this stuff in a transparent way.

The idea is that it doesn't super matter where the proposal is opened, but we should collect related ideas in certain places so we can look at them as a collection of issues and find a coherent solution that addresses them all as much as possible. So I'm gonna close this and reference elm-lang/elm-plans#17 which is collecting a bunch of port problems.

If there is more to say about this idea, we should continue on this thread. The fact that is closed is more about decluttering stuff!