Initial OpenAPI spec - Collection endpoints

[nichwall]

PR Overview

This PR is initial work to get an OpenAPI 3.1 spec added to JSDoc comments in the source code. This work was based off of work initially done by @benonymity and information from @Sapd. The PR includes (most) of the schemas defined in the server/objects/ files based on the current Slate API documentation (which is out of date). I have not validated all of these schemas because I was only focusing on the ones required to get the collections endpoints to work. These other schemas still need to be cleaned up, such as the Minified, Normal, and Extended responses for data types and have better examples given, and I included them because it is a lot easier to edit an existing schema than writing them from scratch (and benonymity had already made those schemas from the Slate API docs, so this should include all of the schemas he worked on).

As per Sapd's initial recommendation, I am using swagger-jsdoc to generate YAML files directly from the source code, done by running node swagger.js in the root directory of the repository. The resulting specification, swagger-output.json, is located in build-docs/. I also have created a very poorly written api_runner.py script that I am using to test API endpoints. It's pretty hacky and I can remove it from the PR, but am including it for initial review to see if a cleaned-up version would be beneficial during this initial work.

I will continue working through updates for each of the endpoints and schemas. If we want to do this a different way that is okay with me, just trying to help get the ball rolling.

Further discussion

To simplify support for multiple API versions, I would like to use a route-based versioning strategy. I think we should provide separate specification files for each version instead of putting everything in the same file with various versions.

At some point, we will need to address documentation translations, as all official documentation is only in English. I am not sure how best to approach this, but it looks like the easiest way will be using overlays similarly to how the server/client strings are managed.

We should implement automated testing for the API spec. Initially, it only needs to validate request/response formatting to ensure the data types match up and conform to the spec. This will be useful during the API rewrite and future database changes as it will alert of data type changes without needing to constantly update tests for different internal data operations.

My current testing procedure

I have written a Python script which connects to a dev server through wiretap (https://pb33f.io/wiretap/), which performs validation of requests and responses against an OpenAPI spec. I have also put wiretap in between the dev client and server to alert me of endpoints that are not in the spec while navigating manually in the client (though right now there are many failures because I've been focusing on collections). This has been working well and wiretap has a Docker image we could use as part of automated testing, but I am not sure how the e2e testing will work in GH Actions so this automated testing would still need to happen in local dev environments right now.

The creator of wiretap has made the vacuum tool to lint OpenAPI specs. This has helped me identify duplicate descriptions or other issues with the generated spec. Both wiretap and vacuum are in active development, so they definitely have some bugs and features missing but are quick to set up and work well enough for my process.

We can switch to something like https://www.npmjs.com/package/chai-openapi-response-validator to integrate with the existing chai test framework, but that means we need to use OpenAPI 3.0 instead of 3.1 and am still not sure how well that testing will work as a GH Action.

Next steps

The only decision that needs to be made before completing the initial OpenAPI spec is whether we want to switch to 3.0 or continue with 3.1.
The following blockers need to be discussed before continuing with a new API version:

• API versioning method - https://www.codemzy.com/blog/nodejs-api-versioning
• Spec formatting - https://blog.treblle.com/documentatuin-in-api-versioning/#how-to-document-multiple-api-versions
• Translations - Multilingual support OAI/OpenAPI-Specification#1740

Additional reading on spec organization: https://blog.liblab.com/why-your-open-api-spec-sucks/

[nichwall]

Performed some schema cleanup and cleaned up some OpenAPI spec linting issues. These schemas have not been validated, but do make those source files shorter. This cleanup does not affect existing endpoints for this PR.

[nichwall]

After continuing work with the library and author endpoints in another branch, I think this method of doing the OpenAPI schema will be a bit too difficult to manage.

What I have done so far is to put the schema definitions in the corresponding objects file, and am then putting the definition for each endpoint above the function in the controller files.

I think there should instead be a single large comment at the top of the corresponding controller files (leaving the objects files unchanged). This is mostly because there is so much dependence between schemas and reusing routes, it is difficult to see what is going on even in the same file with the existing definitions at the top of the file. I would still like to keep everything in the specific file, but just not break it up into different comments throughout the file. Breaking it up may make more sense after everything is written, but right now I think it will make it more difficult to review.

[nichwall]

After additional work and attempts, this PR is not the best way to do things. Even if the method in #2803 is not the solution, the spec in this PR needs to be reworked a bit for 3.0 vs 3.1 and thinking through the file organization.