Decision: Regulation subjects
| Thing | Info |
|---|---|
| Relevant features | Subject pages, search, regulation pages |
| Date started | 2024-05-01 |
| Date finished | |
| Decision status | In progress |
| Summary of outcome | We shouldn't reuse the related resource subject data for annotating the regulations. |
We believe that if we enabled the following interactions, they could help some of our users with policy research:
- When a person looks up related guidance and rules in a regulation sidebar, they can also visit related subjects to see an overview of recent material in related subjects.
- When a person looks at a subject page, they see relevant regulation sections in the list of items (alongside resources).
- When a person searches for some keywords, the regulation sections in the results are annotated with subjects (similar to resources). If they filter the search results by subject, the results include relevant regulations.
Mockup of filtering search results by subject and seeing relevant regulations:
However, we don't have data that answers the question "What are the subjects related to a section?" (same question as "What are the sections related to a subject?").
We do have data that answers a different question: "What are the top subjects associated with the resources related to a section?"
Can we use the answer to "What are the top subjects associated with the resources that are tagged to a section?" to also answer "What are the subjects related to a section?"
If not, is it worth building a way to hand-annotate regulation sections with subjects?
Our users need accurate and comprehensive information for their policy research. If we can't provide information that is accurate and comprehensive, we need to either frame the information carefully so that users have the right expectations or not show the information at all. As an example of careful framing, our statute page include links to SSA.gov but notes it was last updated in 2019, and we observed that at least some users do notice that date.
We have limited time and capacity for new feature development, so we need to focus on features that will have a big impact on making policy research easier.
It could be useful to enable finding regulations by subject because:
- On a combined search page, if you search for some keywords and filter the results by subject, the user could expect to see relevant regulations in the filtered results.
- Some topics are covered in various regulation sections in several parts, so collecting those sections together on a subject page could be helpful.
- Some regulations use language that isn't the everyday term for something - for example, "IMD exclusion" gets 0 search results in regulations, so we provide an imperfect workaround in the form of synonym suggestions:
However, subjects are less necessary for making regulations findable, compared to resources:
- In most cases, the regulations are well-structured with clear section titles.
- Regulations are organized by topic in their own way.
In user research, we've heard more about needing to look up information by subject for statute than regulations.
For the purpose of displaying "top subjects of related documents" in sidebars, our SME is reviewing the preliminary data, and we think we can have confidence in the accuracy of that feature if we limit it to showing subjects tagged to at least X documents (TBD, probably 3 or 5). Assuming we use that kind of threshold, some regulation sections won't display any top subjects, and that's fine.
However, there's a fair bit of imprecision in the data:
- A bunch of documents cover multiple semi-unrelated topics, so they're tagged to multiple subjects and multiple regulation sections. This means that some of those subjects have nothing to do with some of the regulation sections. For example, the CIBs titled "Recent Developments in Medicaid and CHIP Policy", like this one, are a grab bag of topics.
- A bunch of documents cover topics that are related to something in a tagged regulation section but not covered by that regulation. For example, top related subjects for 440.120 include Substance Use Disorder, but the section does not cover that topic explicitly.
It may be ok to have that kind of imprecision in a component labeled "top subjects of related documents", because the reader has enough context to reason about the information.
Subjects tagged to a regulation section wouldn't have that kind of label - they would look just like our hand-annotated subjects on documents - so we need a quite high threshold of confidence there. This data does not seem to be adequate for answering the question "What are the subjects related to a section?".
We could build hand-annotation: our devs could enable annotating regulation sections with subjects, and our SME could do the annotations.
We have so many parts, subparts, and sections that it could take a long time to hand-annotate all of them with subjects, and our SME capacity is limited. There is not much value in less-than-complete annotation, because people expect to get comprehensive results if they look up a subject. For example, if a person looked up EPSDT and saw one of the related sections but not all of them, that could be confusing and reduce trust in eRegs.
We could hypothetically use our future work on semi-automated text classification for internal files (using natural language processing techniques) to make the hand-annotation process faster for regulations too, but we're not expecting to be able to classify documents fully automatically - to ensure accuracy, classification will always require SME review.
We've considered enabling hand-annotation of subjects with related regulation and statute citations, to be displayed as a kind of caption at the top of the subject page (not as an item in the results list). That could serve some of the user needs, but it would be a fair bit of annotation work, and we weren't sure if it was useful enough to be worthwhile.
How long would it take to hand-annotate regulation sections with subjects? Could we use that to generate related sections at the subpart level, or would that need to be hand-annotated too?
We're not sure how we would display regulations on the subject pages - they don't have dates, so by default they would go to the end of the list in alphabetical order with the other undated items.
Should we use the top subjects data for annotating sections, for the purpose of displaying regulations on subject pages?
No, our SME does not think this data is usable for this purpose.
We're working on this.
Could be useful, but we're not sure yet if it's too much work for not enough value.
Maybe, although it's unlikely to be a make-or-break feature for us. Probably more value in figuring out what we can do to help people find what they need in statute.
We did some usability testing for single-column search and subject page designs with the concept that regulations would be annotated with subjects, so we would like to re-test them without that aspect to see if the designs still make enough sense.