External code list usage

[RomanKapitonov]

ElementDefs for lots of elements contain references to external code lists. What is a proper way to utilize those links? For example:

E1271 = t::AN.new(:E1271, "Industry Code"                        , 1, 30,
            s::CodeList.external("508"))

Inspecting an AN referencing an external list just calls a #to_s on CodeList::External. I've found a note in todo with the following content:

"Support for configurable external code sources. The minimal interface should
implement #include?(code). Need to decide how to handle unconfigured external
code sources -- should there be a method to list all the code sources in some
transaction set?"

This is dated 2011, so i believe it has never been implemented since. I would like to either figure out on how to make those external code lists of use in case i missed something, or collect opinions on what you think the best way to implement this with minimal impact in case it's still a pending milestone.

[kputnam]

Recently I was reviewing the entire code base and also noticed this. I would be very happy if someone contributed this feature! If you can wait until middle or end of next week, I will type up some notes. I think some of the structure is there already, some of it is missing (but I had thoughts about it), and the rest of it might be make-up as we go.

The important part will be letting users setup their own code lists from an external source like a database, a CSV file, etc. We don't want people submitting PRs with their code lists. There is a beginning in Config: in your case it would be something like config.code_list.register("508") { ... }. But the ... part is not spec'd and I think we'll need to make other changes for validation to work consistently with how it works for static code lists.

Anyway, I'll take a closer look and post more details sometime next week.

[kputnam]

Hey @RomanKapitonov, sorry this was neglected while working on a larger PR. This was in the back of my mind while working on #164, and here are my thoughts so far:

There are a few parts involved. First is defining some kind of interface for the code list that allows checking for membership. Second is letting users plug their instances in, kind of like how they can plug in their own grammars. Lastly, is how these should be used by stupidedi and/or exposed via the user-level API.

Regarding the interface, I want to be as flexible as possible, because I know of very different kinds of code lists. Some are small and could be held in memory, some are large and might be stored in a database, and others might even be accessible via an API call (like checking if a doctor's NPI number is valid). Some might be validations using a regular expressions or checksums (eg, for SSNs or credit card numbers). I'm also aware some code lists have entries that expire on a given date, or aren't active until a given date.

So maybe the interface would have a method like CodeList::External#at(value, date=nil) which returns Either<String, String>. The left case is an error like Either.error("value 'xyz' is not valid") or Either.error("value 'xyz' is not valid after 2018-01-30"). The right case would be the same kind of value we have for internal code lists, just some string that describes the code like Either.success("an X with a Y and a Z").

Maybe the date defaults to today, or somehow is inferred from some other element(s), but I'm not yet sure how users would specify their own date. I think that might come up, for example, when sending a payment for an order that was sent last month, and you want to ensure you're using last month's set of valid codes.

Regarding how to let users plug their external code lists in, I think there's already stuff in Config, but it might or might not need minor changes as other things come into shape.

Now regarding how this interface would be used by stupidedi, I think there are two cases. Some elements like E26 Country Code refer to one, and only one, external code list. Users could do something like config.code_lists.register("5", MyCodeLists::Countries.new(...)) and validation would work more or less like it does with internal code lists.

The second case is when one element specifies which code list is used to validate another element. For instance, some of the values in E66 Identification Code Qualifier, when used in NM108, control the allowed values in NM109.

For those cases, I think we need to extend SegmentDef and SegmentDef.build to add something similar to SyntaxNote. I don't know what to call it, but it would allow us to indicate for example element 8 specifies the code list for the values in elements 9, 10, 11, or whatever. Then most of the code that uses allowed_values and the internal code lists could be used as a reference to provide similar validation and even printing the value returned by #lookup

I'm sure to be missing some steps, but hopefully this gives some direction. If you or anyone else want to take a stab at it, I'd be happy to review or offer guidance along the way.