jhelvy.com: {surveydown}: An open source, markdown-based survey framework (that doesn't exist yet)

[utterances-bot]

jhelvy.com: {surveydown}: An open source, markdown-based survey framework (that doesn't exist yet)

https://www.jhelvy.com/posts/2023-04-06-markdown-surveys/

[jdimeo]

I built exactly this! I am in the process of trying to open source it with my org.
A lot to say here, but two quick hits:

• We standardize on https://xlsform.org/en/ so we can fully code generate off the YAML + Markdown and it will then be supported by the ODK family of platforms
• We define variables first as "stand alone" ways of measuring something, and then "string them together" into a survey flow. So the survey is modular from the ground up.

[jhelvy]

Oh wow that sounds awesome! That's a main reason why I wrote this post. I wanted to find what else others are already working on. I don't want to re-invent the wheel here, just wanted to outline some ideas for a framework I have in mind. I've had this in mind for years, but never wrote it down before. Please do let me know if you do open source it!

[jdimeo]

A few more notes:

• After beginning to build this framework, we independently discovered another survey/data collection framework geared at lawyers: https://docassemble.org/ Again, a very similar mindset of YAML + Markdown + a syntax language for calculations
• The repo link didn't work? is the repo private?
• I personally am in central VA so we should pow wow some time! (once I get clearance from legal :-))

[jhelvy]

Oh wow docassemble looks very close to what I had in mind. Happy to get on a call or meet in person sometime if you're in DC.

And yes I accidentally had the repo in private mode still. It should be live now: https://github.com/jhelvy/surveydown

The repo isn't much, mostly just the shiny example I discussed in my post. I am also looking at possibly building this with streamlit. As much as I prefer R over python for what I do, an application like this I expect would go a LOT further if it was written in python, so I'm trying to find something to built off of that is python-based. docassemble looks like a strong contender.

[gadenbuie]

This is a cool idea and I'm excited to see what comes from it! Also excited to see what other community projects surface.

A small suggestion: for adding questions I'd recommend using Quarto-style shortcodes, e.g. {{< survey age >}} or {{< question age >}} instead of just {{ age }}. I think you'd probably want to build with Quarto support in mind.

[jhelvy]

@gadenbuie I was thinking about Quarto, but wasn't sure about how to integrate it in. I have to think about the execution flow of how this would all work, which I haven't given too much thought yet.

Assuming I stick with Shiny for the backend, if a user made the survey content using a collection of .qmd files along with a _survey.yml file to control the survey logic and _questions.yml to define the questions, then the compilation steps might be something like:

1. Parse the .qmd files to obtain the overall survey architecture (number of pages, which content is on which page, etc.)
2. Inject the questions from _question.yml as appropriate widgets in the appropriate shortcode locations.
3. Render the .qmd files to obtain the html, complete with widgets now defined in the appropriate locations.
4. Building the corresponding UI and server code to link up the response choices to a data storage object (maybe just a list).

Step 4 would be handled with package functions so the user doesn't have to hand-define any of the UI or server elements ( just like how {shinysurvey} works). The user would just need to provide some basic elements, like the path to the survey folder that contains the .qmd and .yml files, as well as some form of authentication to connect to a database where the survey results will be stored.

There's obviously a lot of complexity here in these different elements, but at this stage I want to keep ideating on the high-level design. By defining out what I want this package to do, I may end up identifying a different approach that works betters. I've been looking at streamlit, for example, as a potential platform for building this in python.

One thing interesting about Quarto is that it may actually work even if I built the package for python instead of R (I think?). I'm obviously far more familiar with R and shiny, but there may be other benefits to using python instead of R for the framework.

Anyway, lots of ideas!

[jhelvy]

I laid out some more thoughts on the overall architecture in this issue in the surveydown repo. I'm still going back and forth on interface - any thoughts anyone has are much appreciated! I want to think through some of the longer-term goals before digging into building anything.