Jaxon dependency maintenance status

[thbar]

Jaxon is used in a couple of places in the library.

The thing is this library has not seen any commit in the last 2.5 years (https://github.com/boudra/jaxon).

I would suggest that we consider moving to something with a better maintenance ultimately, and try to reduce expanding the usage in instructor_ex.

Opening the issue to ensure we are all aware of that!

Related links:

• Streaming support michalmuskala/jason#25
• https://twitter.com/michalmuskala/status/1760717159167078864
• Add json module erlang/otp#8111 (linked above - not merged at time of writing, but supporting streaming if I read quickly)

[thmsmlr]

This is a great callout. I have a local branch with a prototype of implementing JSON Streaming myself.

That being said, this isn't a urgent concern for me as the JSON SPEC is unchanging and jaxon has quite a comprehensive test suite.

For me, the real reason to use change this bit of the library is to change how stream parsing actually works. Currently, when the accumulator is

{ "foo" : "ba

The stream parser only emits {, "foo", : as tokens.

Where as I'd like a stream parser that emits {, "foo", :, "ba so that I can do stream emits within a string as well.

To get this working "properly" with Jaxon was going to be a big pain. There is an alternative hacky solution where we just try parsing by appending a " and see if it's validish. But, that's jank for two reasons. First, is it possible to that appending a " can produce a valid, but different tokenstream? Second, is that although stream parsing is implemeneted as a continuation parser, the continuation fuction is hidden so you can't "fork" it in a low cost way AFAIK.

Anyways, that was more than you bargained for. When we revisit this code, i'm only going to make a change if we can also support streaming partial strings. That's a big feature in other language implementations of instructor.

[thbar]

That being said, this isn't a urgent concern for me as the JSON SPEC is unchanging and jaxon has quite a comprehensive test suite.

100% agreeing ; I was mostly opening the issue to ensure we do not expand the use unknowingly, as I wasn't aware of how aware you would be or not :-)

Anyways, that was more than you bargained for.

I do appreciate the depth of feedback you provided 😄 I will mull over this, and if I have actionable ideas, will not hesitate to contribute on this.

[thmsmlr]

More thoughts on this issue,

This came up again recently with how instructor parses server side events. It turns out we were swallowing useful error messages in the library because of how we were parsing the server side events. And so the use case for having a customizable stream parser reared its head.

For example, server-side event streams can look like this,

data: { "some" : "json"}

data: { "some" : "json"}

data: { "some" : "json"}

and like this,

data: { "some" : "json"}
event: delta
data: { "some" : "json"}
event: delta
data: { "some" : "json"}
event: done

Or in the case of an error, like this

{
    "error" : "some error",
    "code" : ...
}

I have some code that manually handles parsing these three different cases, but this is going to continually be a sore spot for the library until we get a proper parser in there. or at least some code that can accumulate the full stream and add it to the error when it fails to parse so that it is at least easier to debug.

Unfortunately, NimbleParsec doesn't support stream continuation parsing. I've explored writing one myself manually with the help of o3-mini. However, the code is pretty gnarly.

I've so far lost three nights over the last year to this problem. Maybe sometime this year I'll take a stab at writing my own meta compiler like nimbleparsec that supports stream parsing.