Skip to content

Commit

Permalink
API generation script uses Box to download CI artifacts (Qiskit#737)
Browse files Browse the repository at this point in the history
Part of Qiskit#719

This PR modifies the API generation script to download the CI artifacts
stored in the Box folder https://ibm.ent.box.com/folder/246867649571 and
removes the `-a <artifact-url>` argument.

The PR also renames the old `scripts/lib/api/downloadCIArtifacts.ts` to
`scripts/lib/api/apiArtifacts.ts` to make it more generic, and it
removes the associated test. The test file only was related to the `-a`
argument that was removed.

---------

Co-authored-by: Eric Arellano <[email protected]>
  • Loading branch information
arnaucasau and Eric-Arellano authored Feb 1, 2024
1 parent 7439931 commit ce433e9
Show file tree
Hide file tree
Showing 6 changed files with 144 additions and 87 deletions.
47 changes: 36 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -238,28 +238,53 @@ If you're working on our support code in `scripts/`, run `npm run fmt` to automa

To check that formatting is valid without actually making changes, run `npm run check:fmt` or `npm run check`.

## Regenerate the API docs
## Regenerate an existing API docs version

1. Install and configure GitHub CLI: https://docs.github.com/en/github-cli/github-cli/quickstart
2. Choose which documentation you want to regenerate: `qiskit`, `qiskit-ibm-provider`, or `qiskit-ibm-runtime`
3. Determine the current version of the published stable documentation, e.g. at https://github.com/Qiskit/qiskit/releases
4. Find a link to a CI artifact with the project's documentation, e.g at https://github.com/Qiskit/qiskit/suites/17881600359/artifacts/1026798160. To find this:
This is useful when we make improvements to the API generation script.

1. Choose which documentation you want to generate (`qiskit`, `qiskit-ibm-provider`, or `qiskit-ibm-runtime`) and its version.
2. Run `npm run gen-api -- -p <pkg-name> -v <version>`,
e.g. `npm run gen-api -- -p qiskit -v 0.45.0`

If the version is not for the latest stable minor release series, then add `--historical` to the arguments. For example, use `--historical` if the latest stable release is 0.45.\* but you're generating docs for the patch release 0.44.3.

## Generate new API docs

This is useful when new docs content is published, usually corresponding to new releases or hotfixes for content issues. If you're generating a patch release, also see the below subsection for additional steps.

1. Choose which documentation you want to generate (`qiskit`, `qiskit-ibm-provider`, or `qiskit-ibm-runtime`) and its version.
2. Determine the full version, such as by looking at https://github.com/Qiskit/qiskit/releases
3. Download a CI artifact with the project's documentation. To find this:
1. Pull up the CI runs for the stable commit that you want to build docs from. This should not be from a Pull Request
2. Open up the "Details" for the relevant workflow.
- Qiskit: "Documentation / Build (push)"
- Runtime: "CI / Build documentation (push)"
- Provider: "CI / Build documentation (push)"
3. Click the "Summary" page at the top of the left navbar.
4. Scroll down to "Artifacts" and look for the artifact related to documentation, such as `html_docs`.
5. Copy the link by right-clicking on the artifact.
5. Run `npm run gen-api -- -p <pkg-name> -v <version> -a <artifact-url>`,
e.g. `npm run gen-api -- -p qiskit -v 0.45.0 -a https://github.com/Qiskit/qiskit/suites/17881600359/artifacts/1026798160`
6. When opening your PR, include the CLI arguments you used. That helps us to know exactly how the docs have been generated over time.

If the version is not for the latest stable minor release series, then add `--historical` to the arguments. For example, use `--historical` if the latest stable release is 0.45.\* but you're generating docs for the patch release 0.44.3.
5. Download the artifact by clicking on its name.
4. Rename the downloaded zip file with its version number, e.g. `0.45.zip` for an artifact from `qiskit v0.45.2`.
5. Upload the renamed zip file to https://ibm.ent.box.com/folder/246867649571
6. Share the file by clicking the `Copy shared link` button
7. Select `People with the link` and go to `Link Settings`.
8. Under `Link Expiration` select `Disable Shared Link on` and set an expiration date of ~10 years into the future.
9. Copy the direct link at the end of the `Shared Link Settings` tab.
10. Modify the `scripts/lib/api/api-html-artifacts.json` file adding the new versions with the direct link from step 9.
11. Run `npm run gen-api -- -p <pkg-name> -v <version>`,
e.g. `npm run gen-api -- -p qiskit -v 0.45.0`

In case you want to save the current version and convert it into a historical one, you can run `npm run make-historical -- -p <pkg-name>` beforehand.

### Generate patch releases

For example, if the current docs are for 0.45.2 but you want to generate 0.45.3.

Follow the same process above for new API docs, other than:

- When uploading the artifact, overwrite the existing file with the new one. Be careful to follow the above steps to change "Link Expiration"!

If the version is not for the latest stable minor release series, then add `--historical` to the arguments. For example, use `--historical` if the latest stable release is 0.45.\* but you're generating docs for the patch release 0.44.3.

# How to write the documentation

We use [MDX](https://mdxjs.com), which is like normal markdown but adds extensions for custom components we have.
Expand Down
35 changes: 35 additions & 0 deletions scripts/api-html-artifacts.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
{
"qiskit": {
"0.45": "https://ibm.box.com/shared/static/e2rwbrztml4dg6lkk1w3e0kmevkw9w7s.zip",
"0.44": "https://ibm.box.com/shared/static/yudjd08saugb3z9stciz4zxhpkroqrv4.zip",
"0.43": "https://ibm.box.com/shared/static/odcdrizljwpcz96upbu8umb5xpfvf73h.zip",
"0.42": "https://ibm.box.com/shared/static/muj1v0m4x36fuqyqo22wknexj9cq8yfv.zip",
"0.41": "https://ibm.box.com/shared/static/2j2qw33dgy5swf9vno0tfpq8ahey14uu.zip",
"0.40": "https://ibm.box.com/shared/static/6j9mbwbeqx0kfi20zl22hzp6524kvqz8.zip",
"0.39": "https://ibm.box.com/shared/static/za2u7phhpyd6lrsalv48xsxhz1ud17rd.zip",
"0.38": "https://ibm.box.com/shared/static/rcyb5ugvgjc6gpgb5hkdedz5r5buashu.zip",
"0.37": "https://ibm.box.com/shared/static/4nfrhcnu5ixod1i5ct1fps2qpoyfmgnb.zip",
"0.36": "https://ibm.box.com/shared/static/eekpk9r8kawwxr1bfaswrcc073mien1j.zip",
"0.35": "https://ibm.box.com/shared/static/dkqlxxllnhrq4yyae9fqlifab36fq2av.zip",
"0.33": "https://ibm.box.com/shared/static/kmgdgi0imgam3o7jzumyjhwdz00i9y5w.zip",
"0.32": "https://ibm.box.com/shared/static/tzkb58t2vgzz21i6ydg7r9g4m4995waq.zip",
"0.31": "https://ibm.box.com/shared/static/d4m9d1f4uaq95ovpqo9yot1mnm1jnofw.zip",
"0.30": "https://ibm.box.com/shared/static/loueznt45qyoo925gwt6cb5n1a0lp4o0.zip",
"0.29": "https://ibm.box.com/shared/static/uc8s5lcrmxuy23vkb2bkn8y930m533rk.zip",
"0.28": "https://ibm.box.com/shared/static/656e5bsov004vpnlmc9z40q718eq0ze2.zip",
"0.27": "https://ibm.box.com/shared/static/y2lb96gl2v32si02xgbswmjc06vfvfje.zip",
"0.26": "https://ibm.box.com/shared/static/5wlx2c8wr5xka5buzmlf1mqmz0zl600p.zip",
"0.25": "https://ibm.box.com/shared/static/h9dw4iq19f6nz6vp0egpoyjhxqnx7ot5.zip",
"0.24": "https://ibm.box.com/shared/static/pmcnjymbenwhwa3psxqsptvexs7hwrpd.zip",
"0.19": "https://ibm.box.com/shared/static/ov1hqihl0hlsai31dwyjyjj06k86k2nt.zip"
},
"qiskit-ibm-provider": {
"0.7": "https://ibm.box.com/shared/static/o5tjrdefdz72yi35nw09l9zbjjjecips.zip"
},
"qiskit-ibm-runtime": {
"0.17": "https://ibm.box.com/shared/static/5r7r2x65bst3hdtcyuowwfvno5o23xw0.zip",
"0.16": "https://ibm.box.com/shared/static/xbtjc270jc2uu3s8tp7tqn8o9pckl37i.zip",
"0.15": "https://ibm.box.com/shared/static/j9wiuo9mga3lwihhqy9sdeqtsr4taanm.zip",
"0.14": "https://ibm.box.com/shared/static/bhv1xl2pid74qsanmphx3zm49cup6owv.zip"
}
}
17 changes: 8 additions & 9 deletions scripts/commands/updateApiDocs.ts
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ import { dedupeHtmlIdsFromResults } from "../lib/api/dedupeHtmlIds";
import { Pkg } from "../lib/api/Pkg";
import { zxMain } from "../lib/zx";
import { pathExists, getRoot, rmFilesInFolder } from "../lib/fs";
import { downloadCIArtifact } from "../lib/api/downloadCIArtifacts";
import { downloadCIArtifact } from "../lib/api/apiArtifacts";
import {
addNewReleaseNotes,
generateReleaseNotesIndex,
Expand All @@ -47,7 +47,7 @@ interface Arguments {
package: string;
version: string;
historical: boolean;
artifact: string;
skipDownload: boolean;
}

const readArgs = (): Arguments => {
Expand All @@ -71,12 +71,11 @@ const readArgs = (): Arguments => {
default: false,
description: "Is this a prior release?",
})
.option("artifact", {
alias: "a",
type: "string",
demandOption: true,
.option("skip-download", {
type: "boolean",
default: false,
description:
"The URL for the CI artifact to download. Must be from GitHub Actions.",
"Rather than downloading the artifact from Box, reuse what is already downloaded. This can save time, but it risks using an outdated version of the docs.",
})
.parseSync();
};
Expand All @@ -100,10 +99,10 @@ zxMain(async () => {
);

const artifactFolder = pkg.ciArtifactFolder();
if (await pathExists(artifactFolder)) {
if (args.skipDownload && (await pathExists(`${artifactFolder}/artifact`))) {
console.log(`Skip downloading sources for ${pkg.name}:${pkg.version}`);
} else {
await downloadCIArtifact(pkg.name, args.artifact, artifactFolder);
await downloadCIArtifact(pkg, artifactFolder);
}

const outputDir = pkg.outputDir(`${getRoot()}/docs`);
Expand Down
65 changes: 65 additions & 0 deletions scripts/lib/api/apiArtifacts.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
// This code is a Qiskit project.
//
// (C) Copyright IBM 2023.
//
// This code is licensed under the Apache License, Version 2.0. You may
// obtain a copy of this license in the LICENSE file in the root directory
// of this source tree or at http://www.apache.org/licenses/LICENSE-2.0.
//
// Any modifications or derivative works of this code must retain this
// copyright notice, and modified files need to carry a notice indicating
// that they have been altered from the originals.

import { $ } from "zx";
import { createWriteStream } from "node:fs";
import { finished } from "stream/promises";
import { Readable } from "stream";
import fs from "fs";

import { pathExists, getRoot } from "../fs";
import { mkdirp } from "mkdirp";
import { Pkg } from "./Pkg";

async function downloadFromBox(
pkgName: string,
artifactUrl: string,
destination: string,
) {
const response = await fetch(artifactUrl);
if (response.ok) {
const stream = createWriteStream(destination);
await finished(Readable.fromWeb(response.body as any).pipe(stream));
} else {
throw new Error(`Error downloading the ${pkgName} artifact from Box.`);
}
}

export async function downloadCIArtifact(pkg: Pkg, artifactFolder: string) {
if (await pathExists(artifactFolder)) {
await $`rm -rf ${artifactFolder}/*`;
} else {
await mkdirp(artifactFolder);
}

const artifactJson = JSON.parse(
fs.readFileSync(`${getRoot()}/scripts/api-html-artifacts.json`, "utf-8"),
);

if (!(`${pkg.versionWithoutPatch}` in artifactJson[`${pkg.name}`])) {
throw new Error(
`Package ${pkg.name} version ${pkg.versionWithoutPatch} doesn't have an artifact stored. You can add one to https://ibm.ent.box.com/folder/246867452622
following the steps detailed in the \`Generate the API docs\` section on the repo's README. If you are not an IBMer with access to the Box folder,
you can ask in your pull request for a maintainer to help you. In the meantime, you can use another URL in api-html-artifacts.json, such as GitHub or
even localhost for a server you start up; the URL needs to result in downloading the zip file.`,
);
}
const artifactUrl = artifactJson[`${pkg.name}`][`${pkg.versionWithoutPatch}`];

await downloadFromBox(
pkg.name,
artifactUrl,
`${artifactFolder}/artifact.zip`,
);

await $`unzip -qqo ${artifactFolder}/artifact.zip -d ${artifactFolder}/artifact`;
}
29 changes: 0 additions & 29 deletions scripts/lib/api/downloadCIArtifacts.test.ts

This file was deleted.

38 changes: 0 additions & 38 deletions scripts/lib/api/downloadCIArtifacts.ts

This file was deleted.

0 comments on commit ce433e9

Please sign in to comment.