299 improve schema reference #370

duncandewhurst · 2022-11-11T03:21:12Z

Closes Move content describing the use of the schema from the worked example to the schema reference #299

TO-DO:

duncandewhurst · 2022-11-13T20:35:45Z

@jpmckinney please could you check that you're happy with the direction of this PR?

I reused a script that I developed for another project to generate an improved schema reference page. You can preview it here: https://standard.open-contracting.org/staging/infrastructure/299-improve-schema-reference/en/reference/schema/. The main changes are to:

Replace 'components' terminology with 'sub-schema' as suggested in Eliminate language of "building blocks" standard#1347
Reduce reference table length by collapsing all fields that reference a sub-schema
Add the definition/description of each sub-schema
Add a list of links to the fields that reference each sub-schema
Add an example for each sub-schema

The script can be run as part of pre-commit to keep the schema reference documentation up to date with changes to the schema. It will preserve any manually authored content at the top of the page and under the heading for each sub-schema.

duncandewhurst · 2022-11-21T00:32:24Z

@jpmckinney please could you check that you're happy with the direction of this PR?

@jpmckinney just checking that you saw this request - thanks!

jpmckinney

I ran the command, and each time it keeps adding more content...

Edit: Fixed locally.

jpmckinney · 2022-12-01T18:49:36Z

manage.py

+        # Add description
+        definition["content"].extend([
+            f"`{defn}` is defined as:\n\n",
+            "```{jsoninclude-quote} ../../schema/project-level/project-schema.json\n",


sphinxcontrib-opencontracting already had https://sphinxcontrib-opencontracting.readthedocs.io/en/latest/#field-description before this was added to sphinxcontrib-opendataservices.

Doh! Which would you prefer to use?

I missed this earlier comment - I prefer field-description.

Done in e65bbf5

manage.py

jpmckinney · 2022-12-01T19:22:16Z

I prefer https://sphinx-inline-tabs.readthedocs.io/en/latest/ (works without JS, easier to customize) to sphinx-tabs.

jpmckinney · 2022-12-01T19:32:47Z

The command is failing because these don't exist

/projects/0/forecasts/0/observations/0/value
/projects/0/metrics/0/observations/0/value
/projects/0/contractingProcesses/0/summary/modifications/0/oldContractValue (needs to be index 2)
/projects/0/contractingProcesses/0/summary/modifications/0/newContractValue (needs to be index 2)

jpmckinney · 2022-12-01T20:30:03Z

I got make to run.

I added empty value to forecasts and metrics for now.
schema.md needs to be hand-edited to correct the indices for oldContractValue and newContractValue.

Note this diff after running pre-commit:

--- a/docs/_static/i18n.csv
+++ b/docs/_static/i18n.csv
@@ -14,7 +14,7 @@ sector,Project sector,False,
 purpose,Project purpose,True,
 additionalClassifications,Additional classifications,False,
 additionalClassifications,Classification,False,
-additionalClassifications/scheme,Scheme,True,
+additionalClassifications/scheme,Scheme,False,
 additionalClassifications/id,ID,True,
 additionalClassifications/description,Description,True,
 additionalClassifications/uri,URI,False,

jpmckinney · 2022-12-01T20:34:39Z

I think we can extract the "this is referenced by" logic to a directive in sphinxcontrib-opencontracting, as suggested in open-contracting/standard#1075 (comment)

Otherwise, I think the rest of the logic can't be generalized, as it's mostly code to add directives in a way that's fairly specific to this project (e.g. other projects might prefer a different order or set of directives).

jpmckinney · 2022-12-01T20:36:27Z

This might be for a follow-up PR, but I think it'd be nicer to put definitions on separate pages, as suggested in open-contracting/standard#1075 (comment)

duncandewhurst · 2022-12-01T21:08:28Z

Thanks for the feedback! I take the detailed review to mean that you are happy with the direction of the PR so I will work on your comments and the checklist in the PR description.

jpmckinney · 2022-12-01T21:22:42Z

Yup! Direction is good.

duncandewhurst · 2022-12-01T21:54:52Z

I prefer https://sphinx-inline-tabs.readthedocs.io/en/latest/ (works without JS, easier to customize) to sphinx-tabs.

Should we use https://sphinx-design.readthedocs.io/en/rtd-theme/tabs.html instead?

jpmckinney · 2022-12-01T23:02:09Z

Ah, yes, even better.

duncandewhurst · 2023-02-15T01:40:03Z

Noting that I've enabled the colon_fence syntax extension as recommended in https://sphinx-design.readthedocs.io/en/rtd-theme/get_started.html. However, that does now mean there's a mix of colon fences and backtick fences.

duncandewhurst · 2023-02-15T02:38:24Z

I think we can extract the "this is referenced by" logic to a directive in sphinxcontrib-opencontracting, as suggested in open-contracting/standard#1075 (comment)

@jpmckinney do you want to do that before this PR is merged or are you happy for it to live here for now?

jpmckinney · 2023-02-15T13:58:13Z

Noting that I've enabled the colon_fence syntax extension as recommended

This is mainly recommended so that Markdown syntax highlighters will highlight the Markdown inside the fence (when using backticks, it highlights the whole thing as if it were a code block).

If we don't need that highlighting, it'd be simpler to not enable the extension.

@jpmckinney do you want to do that before this PR is merged or are you happy for it to live here for now?

We can just create/update issues to do it later.

duncandewhurst · 2023-03-21T02:09:11Z

Review text in docs/guidance/example.md to identify text that should be moved to docs/reference/schema.md or to field descriptions

Consider removing the worked example in favour of the examples in docs/reference/schema.md

@jpmckinney having reviewed the content of docs/guidance/example.md, it largely repeats content that is already available in either the schema reference page or in the worked example file itself. Now that the examples are integrated into the schema reference documentation, I think that we should reduce repetition by:

Moving the content from https://standard.open-contracting.org/staging/infrastructure/299-improve-schema-reference/en/guidance/example/#overview to the project section of the schema reference page
Removing the content from https://standard.open-contracting.org/staging/infrastructure/299-improve-schema-reference/en/guidance/example/#sections onwards
Updating the remaining content as follows:
- Expand the narrative summary in the second sentence to describe more of the key features of the worked example file, e.g. the nature of the contracting processes, the variations between the budget and completion values etc.
- Link to the schema reference page as the place to view the examples in context of the structure of the schema

Does that sound good?

jpmckinney · 2023-03-21T15:41:20Z

I'm having trouble visualizing what the new example.md page will look like, since if Overview is moved and Sections is removed, I'm not sure what content is left to update – is it just the content at the start of the page?

duncandewhurst · 2023-03-21T20:59:03Z

I was thinking that it would still be useful for the worked example to have a separate page from the schema reference, which could just be the content that appears before the overview heading with the updates suggested above:

jpmckinney · 2023-03-21T22:10:49Z

Aha, okay - sounds fine to me!

duncandewhurst · 2023-03-23T03:02:22Z

@jpmckinney this is now ready for review with the exception of the build errors, which relate to the links below. Any idea how to get these to work?

infrastructure/docs/guidance/example.md

Lines 11 to 13 in 22529f1

    
           * [`forecasts`](../reference/schema.md#project-schema.json,,forecasts) and [`metrics`](../reference/schema.md#project-schema.json,,metrics) that describe planned and actual physical progress 
        
           * [`modifications`](../reference/schema.md#project-schema.json,,metrics) that describe changes to the duration, scope and value of contracting processes 
        
           * [`completion`](../reference/schema.md#project-schema.json,,completion) data, describing the final end date, value and scope of the project.

I ended up restructuring the worked example page a little so that is better set up for us to add other examples in the future. Happy to hear your feedback!

jpmckinney

I committed the fix for the links (drop the part before project-schema.json).

I made a few comments, but haven't read the bulk of the changes (i.e. example.md, schema.md, and manage.py).

We're at a retreat next week, so I'll try to get back to this when I return

docs/conf.py

jpmckinney · 2023-03-25T01:51:01Z

manage.py

+        # Add description
+        definition["content"].extend([
+            f"`{defn}` is defined as:\n\n",
+            "```{jsoninclude-quote} ../../schema/project-level/project-schema.json\n",


I missed this earlier comment - I prefer field-description.

Co-authored-by: James McKinney <[email protected]>

duncandewhurst · 2023-03-29T01:36:24Z

Thanks! I've made those updates.

duncandewhurst · 2023-04-19T22:19:59Z

@jpmckinney just checking if you would have a chance to complete your review over the next week or two? Thanks!

jpmckinney · 2023-04-23T19:44:59Z

In the next week or two, yes!

jpmckinney

I'll commit my Python documentation change. I haven't reviewed the code, but I figure it works, and we'll probably change it if needed when we cover open-contracting/sphinxcontrib-opencontracting#33

manage.py

duncandewhurst added 6 commits November 11, 2022 15:52

requirements.txt: Upgrade sphinxcontrib-opendataservices

d3b0793

manage.py: Add a pre-commit script to update schema reference docs

b9626d4

docs/examples: reorder contracting processes, add missing fields

76400b5

docs/reference/schema.md: Refactor, add definitions and examples

389a54d

docs/reference/schema.md: Use Sphinx Tabs for schema tables and examples

819f9b2

manage.py: Fix lint errors

3dacc5b

duncandewhurst mentioned this pull request Nov 21, 2022

Add pull request template #374

Merged

8 tasks

jpmckinney reviewed Dec 1, 2022

View reviewed changes

Merge 0.9-dev into 299-improve-schema-reference

d634dac

feat: Fix manage.py and update requirements to run make

5268c14

Merge branch '0.9-dev' into 299-improve-schema-reference

d0c6cd7

jpmckinney and others added 2 commits December 2, 2022 00:43

Merge 0.9-dev into 299-improve-schema-reference

b0099cc

docs/reference/schema.md: Use sphinx-design tabs

c558d19

duncandewhurst added 2 commits February 15, 2023 14:55

docs/reference: Update introductory text

bf176e0

manage.py: Handle paths that don't appear in the example data

d222951

manage.py: Fix lint errors

ba3c3c9

duncandewhurst added 2 commits March 21, 2023 14:08

manage.py: Fix lint errors

5b38dcc

docs/reference/schema.md: Reduce repetition of content in the schema

6d73eeb

This was referenced Mar 21, 2023

Schema reference documentation: Put definitions on separate pages #384

Open

Implement a new directive to list where a sub-schema is referenced from open-contracting/sphinxcontrib-opencontracting#33

Open

duncandewhurst mentioned this pull request Mar 22, 2023

Review schema for classes that can be inlined to tidy up the reference page Open-Telecoms-Data/open-fibre-data-standard#143

Open

duncandewhurst added 7 commits March 23, 2023 15:32

docs: Integrate worked example and schema reference

189659a

Don't use colon fence extension

e29c23c

pull_request_template.md: Update merge checklist

1f023dc

Update changelog

8ed9a10

docs/reference/changelog.md: Fix incorrect PR number

42299b1

Merge branch '0.9-dev' into 299-improve-schema-reference

55a2848

docs/guidance/example.md: Add links to field anchors

22529f1

duncandewhurst marked this pull request as ready for review March 23, 2023 03:02

duncandewhurst requested a review from jpmckinney March 23, 2023 03:02

chore: Fix broken refs

ec0041e

jpmckinney reviewed Mar 25, 2023

View reviewed changes

duncandewhurst and others added 2 commits March 29, 2023 14:29

Update docs/conf.py

1ed9476

Co-authored-by: James McKinney <[email protected]>

docs/reference/schema.md: Use field-description directive

e65bbf5

jpmckinney approved these changes Apr 26, 2023

View reviewed changes

manage.py Outdated Show resolved Hide resolved

docs: Fix manage.py docstring

dbd0f97

jpmckinney merged commit f520f8b into 0.9-dev Apr 26, 2023

jpmckinney deleted the 299-improve-schema-reference branch April 26, 2023 01:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

299 improve schema reference #370

299 improve schema reference #370

duncandewhurst commented Nov 11, 2022 •

edited

Loading

duncandewhurst commented Nov 13, 2022

duncandewhurst commented Nov 21, 2022

jpmckinney left a comment •

edited

Loading

jpmckinney Dec 1, 2022

duncandewhurst Feb 15, 2023

jpmckinney Mar 25, 2023

duncandewhurst Mar 29, 2023

jpmckinney commented Dec 1, 2022 •

edited by duncandewhurst

Loading

jpmckinney commented Dec 1, 2022 •

edited

Loading

jpmckinney commented Dec 1, 2022

jpmckinney commented Dec 1, 2022

jpmckinney commented Dec 1, 2022 •

edited

Loading

duncandewhurst commented Dec 1, 2022

jpmckinney commented Dec 1, 2022

duncandewhurst commented Dec 1, 2022 •

edited

Loading

jpmckinney commented Dec 1, 2022

duncandewhurst commented Feb 15, 2023

duncandewhurst commented Feb 15, 2023

jpmckinney commented Feb 15, 2023

duncandewhurst commented Mar 21, 2023 •

edited

Loading

jpmckinney commented Mar 21, 2023

duncandewhurst commented Mar 21, 2023

jpmckinney commented Mar 21, 2023

duncandewhurst commented Mar 23, 2023

jpmckinney left a comment

jpmckinney Mar 25, 2023

duncandewhurst commented Mar 29, 2023

duncandewhurst commented Apr 19, 2023

jpmckinney commented Apr 23, 2023

jpmckinney left a comment

299 improve schema reference #370

299 improve schema reference #370

Conversation

duncandewhurst commented Nov 11, 2022 • edited Loading

duncandewhurst commented Nov 13, 2022

duncandewhurst commented Nov 21, 2022

jpmckinney left a comment • edited Loading

Choose a reason for hiding this comment

jpmckinney Dec 1, 2022

Choose a reason for hiding this comment

duncandewhurst Feb 15, 2023

Choose a reason for hiding this comment

jpmckinney Mar 25, 2023

Choose a reason for hiding this comment

duncandewhurst Mar 29, 2023

Choose a reason for hiding this comment

jpmckinney commented Dec 1, 2022 • edited by duncandewhurst Loading

jpmckinney commented Dec 1, 2022 • edited Loading

jpmckinney commented Dec 1, 2022

jpmckinney commented Dec 1, 2022

jpmckinney commented Dec 1, 2022 • edited Loading

duncandewhurst commented Dec 1, 2022

jpmckinney commented Dec 1, 2022

duncandewhurst commented Dec 1, 2022 • edited Loading

jpmckinney commented Dec 1, 2022

duncandewhurst commented Feb 15, 2023

duncandewhurst commented Feb 15, 2023

jpmckinney commented Feb 15, 2023

duncandewhurst commented Mar 21, 2023 • edited Loading

jpmckinney commented Mar 21, 2023

duncandewhurst commented Mar 21, 2023

jpmckinney commented Mar 21, 2023

duncandewhurst commented Mar 23, 2023

jpmckinney left a comment

Choose a reason for hiding this comment

jpmckinney Mar 25, 2023

Choose a reason for hiding this comment

duncandewhurst commented Mar 29, 2023

duncandewhurst commented Apr 19, 2023

jpmckinney commented Apr 23, 2023

jpmckinney left a comment

Choose a reason for hiding this comment

duncandewhurst commented Nov 11, 2022 •

edited

Loading

jpmckinney left a comment •

edited

Loading

jpmckinney commented Dec 1, 2022 •

edited by duncandewhurst

Loading

jpmckinney commented Dec 1, 2022 •

edited

Loading

jpmckinney commented Dec 1, 2022 •

edited

Loading

duncandewhurst commented Dec 1, 2022 •

edited

Loading

duncandewhurst commented Mar 21, 2023 •

edited

Loading