Decision: Regulation subjects
| Thing | Info |
|---|---|
| Relevant features | Subject pages and search |
| Date started | 2024-05-01 |
| Date finished | |
| Decision status | Draft |
| Summary of outcome |
We believe that if we enabled the following interactions, they could help some of our users with policy research:
- When a person looks at a subject page, they see relevant regulation sections in the list of items (alongside resources).
- When a person searches for some keywords, the regulation sections in the results are annotated with subjects (similar to resources). If they filter the search results by subject, the results include relevant regulations.
We can't do this right now because we don't have data that answers the questions "What are the subjects related to a section?" or "What are the sections related to a subject?"
We do have data that answers a different question: "What are the top subjects associated with the resources that are tagged to a section?"
Can we use the answer to "What are the top subjects associated with the resources that are tagged to a section?" to also answer "What are the subjects related to a section?"
If not, is it worth building a way to annotate regulation sections with subjects?
Our users need accurate and comprehensive information for their policy research. If we can't provide information that is accurate and comprehensive, we need to either frame the information carefully so that users have the right expectations or not show the information at all. (As an example of careful framing, our statute page include links to SSA but notes it was last updated in 2019, and we observed that at least some users do notice that date.)
We have limited time and capacity for new feature development, so we need to focus on features that will have a big impact on making policy research easier.
It could be useful to enable finding regulations by subject because:
- On a combined search page, if you search for some keywords and filter the results by subject, you might expect to still see relevant regulations in the results.
- Some topics are covered in various regulation sections in several parts.
- Some regulations use language that isn't the everyday term for something - for example, "IMD exclusion" gets 0 search results in regulations, so we provide an imperfect workaround in the form of synonym suggestions:
However, subjects are less necessary for making regulations findable, compared to resources:
- In most cases, the regulations are well-structured with clear section titles.
- Regulations are organized by topic in their own way.
We've heard more of this kind of user need (looking up information by subject) for statute citations than regulations.
For the purpose of displaying "top subjects of related documents" in sidebars, our SME is still reviewing the data, but we think we can have reasonable confidence in this feature being helpful if we limit it to showing subjects tagged to at least X documents (TBD, probably 5ish).
Assuming we use that kind of threshold, some regulation sections won't display any top subjects, and that's fine.
We could build hand-annotation: our devs could enable annotating regulation sections with subjects, and our SME could do the annotations.
We have so many parts, subparts, and sections that it would take a long time to hand-annotate all of them with subjects, and our SME capacity is limited. There is not much value in less-than-complete annotation, because people expect to get reasonably comprehensive results if they look up a subject. For example, if a person looked up EPSDT and saw one of the related sections but not all of them, that could be confusing and reduce trust in eRegs.
We could hypothetically use our future work on semi-automated text classification for internal files (using natural language processing techniques) to make the hand-annotation process faster for regulations too, but we're not expecting to be able to classify documents fully automatically - to ensure accuracy, classification will always require SME review.
We've considered enabling hand-annotation of subjects with related regulation and statute citations, to be displayed as a kind of caption at the top of the subject page (not as an item in the results list). That could serve some of the user needs, but it would be a fair bit of annotation work, and we weren't sure if it was useful enough to be worthwhile.
We're not sure how we would display regulations on the subject pages - they don't have dates, so by default they would go to the end of the list in alphabetical order with the other undated items.
Our SME does not think this data is usable for this purpose. [Need to add more detail here about why]
Too much work for not enough value.
Maybe, although it's unlikely to be a make-or-break feature for us. Probably more value in figuring out what we can do to help people find what they need in statute.