Skip to content

Commit

Permalink
Virtual Packages and SUSHI Compatibility Changes
Browse files Browse the repository at this point in the history 
Various changes to better support the needs of SUSHI while remaining neutral for other environments and applications. Including:
* VirtualPackage interface for pseudo packages not in the registry or local FHIR cache
* DiskBased and InMemory implementations of Virtual Package
* Support for loading VirtualPackages in the BasePackageLoader
* Support for '|version' when searching by key
* New exportDB function in SQLJSPackageDB and BasePackageLoader for debugging purposes
* New --export flag in CLI app to export a SQLite DB file from the loaded packages
* Added sort to find functions' options, with initial ByLoadOrder and ByType implementations (the original type argument is now just a filter, no longer affecting sort)
* Added resolveVersion function to RegistryClient interface and corresponding implementations
* Moved LRU cache from DiskBasedPackageCache to BasePackageLoader
  • Loading branch information
@cmoesel
cmoesel committed Nov 5, 2024
1 parent 4c1ebef commit ab29c50
Show file tree
Hide file tree
Showing 58 changed files with 2,833 additions and 610 deletions.
19 changes: 12 additions & 7 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -43,12 +43,13 @@ Usage: fpl install <fhirPackages...> [options]
download and unzip specified FHIR packages

Arguments:
fhirPackages list of FHIR packages to load using the format packageId#packageVersion or packageId@packageVersion
fhirPackages list of FHIR packages to load using the format packageId#packageVersion or packageId@packageVersion

Options:
-c, --cachePath <dir> where to save packages to and load definitions from (default is the local [FHIR cache](https://confluence.hl7.org/pages/viewpage.action?pageId=66928417#FHIRPackageCache-Location))
-d, --debug output extra debugging information
-h, --help display help for command
-c, --cachePath <dir> where to save packages to and load definitions from (default is the local FHIR cache)
-d, --debug output extra debugging information
-e, --export export a SQLite DB file with data from the loaded packages
-h, --help display help for command
```

General information about any command can be found with `fpl --help`:
Expand Down Expand Up @@ -78,6 +79,7 @@ FHIR Package Loader can be used as a library to download FHIR packages, query th
```ts
export interface PackageLoader {
loadPackage(name: string, version: string): Promise<LoadStatus>;
loadVirtualPackage(pkg: VirtualPackage): Promise<LoadStatus>;
getPackageLoadStatus(name: string, version: string): LoadStatus;
findPackageInfos(name: string): PackageInfo[];
findPackageInfo(name: string, version: string): PackageInfo | undefined;
Expand All @@ -101,6 +103,7 @@ The [default PackageLoader](src/loader/DefaultPackageLoader.ts) implementation p
* the standard FHIR registry is used (`packages.fhir.org`) for downloading published packages, falling back to `packages2.fhir.org` when necessary
* unless an `FPL_REGISTRY` environment variable is defined, in which case its value is used as the URL for an NPM registry to use _instead_ of the standard FHIR registry
* the `build.fhir.org` build server is used for downloading _current_ builds of packages
* a 500-item LRU in-memory cache is used to minimize repeated disk reads for resource files

To instantiate the default `PackageLoader`, import the asynchronous `defaultPackageLoader` function and invoke it, optionally passing in an `options` object with a log method to use for logging:

Expand All @@ -116,9 +119,7 @@ if (status !== LoadStatus.LOADED) {
}
```

To instantiate the default `PackageLoader` with a set of standalone JSON or XML resources that should be pre-loaded, use the `defaultPackageLoaderWithLocalResources` function instead, passing in an array of file paths to folders containing the resources to load.

For more control over the `PackageLoader`, use the [BasePackageLoader](src/loader/BasePackageLoader.ts). This allows you to specify the [PackageDB](src/db), [PackageCache](src/cache), [RegistryClient](src/registry), and [CurrentBuildClient](src/current) you wish to use. FHIRPackageLoader comes with implementations of each of these, but you may also provide your own implementations that adhere to the relevant interfaces.
For more control over the `PackageLoader`, use the [BasePackageLoader](src/loader/BasePackageLoader.ts). This allows you to specify the [PackageDB](src/db), [PackageCache](src/cache), [RegistryClient](src/registry), and [CurrentBuildClient](src/current) you wish to use FHIRPackageLoader comes with implementations of each of these, but you may also provide your own implementations that adhere to the relevant interfaces. The BasePackageLoader also allows you to configure the size of the in-memory LRU resource cache.

### PackageLoader Functions

Expand All @@ -128,6 +129,10 @@ The `PackageLoader` interface provides the following functions:

Loads the specified package version. The version may be a specific version (e.g., `1.2.3`), a wildcard patch version (e.g., `1.2.x`), `dev` (to indicate the local development build in your FHIR cache), `current` (to indicate the current master/main build), `current$branchname` (to indicate the current build on a specific branch), or `latest` (to indicate the most recent published version). Returns the [LoadStatus](src/loader/PackageLoader.ts).

#### `loadVirtualPackage(pkg: VirtualPackage): Promise<LoadStatus>`

Loads a resources from a passed in implementation of the [VirtualPackage](src/virtual/VirtualPackage.ts) interface. This allows for "virtual" packages that do not come from a registry nor are stored in the local FHIR package cache. The [DiskBasedVirtualPackage](src/virtual/DiskBasedVirtualPackage.ts) implementation allows resources from arbitrary file paths (folders or direct files) to be loaded as a package. The [InMemoryVirtualPackage](src/virtual/InMemoryVirtualPackage.ts) implementation allows resources in a Map to be loaded as a package. Developers may also provide their own implementation of the VirtualPackage interface. Returns the [LoadStatus](src/loader/PackageLoader.ts).

#### `getPackageLoadStatus(name: string, version: string): LoadStatus`

Gets the [LoadStatus](src/loader/PackageLoader.ts) for the specified package version. The returned value will be `LoadStatus.LOADED` if it is already loaded, `LoadStatus.NOT_LOADED` if it has not yet been loaded, or `LoadStatus.FAILED` if it was attempted but failed to load. This function supports specific versions (e.g. `1.2.3`), `dev`, `current`, and `current$branchname`. It does _not_ support wildcard patch versions (e.g., `1.2.x`) nor does it support the `latest` version.
Expand Down
12 changes: 11 additions & 1 deletion src/app.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ async function install(fhirPackages: string[], options: OptionValues) {
const SQL = await initSqlJs();
const packageDB = new SQLJSPackageDB(new SQL.Database());
const fhirCache = options.cachePath ?? path.join(os.homedir(), '.fhir', 'packages');
const packageCache = new DiskBasedPackageCache(fhirCache, [], { log });
const packageCache = new DiskBasedPackageCache(fhirCache, { log });
const registryClient = new DefaultRegistryClient({ log });
const buildClient = new BuildDotFhirDotOrgClient({ log });
const loader = new BasePackageLoader(packageDB, packageCache, registryClient, buildClient, {
Expand All @@ -48,6 +48,15 @@ async function install(fhirPackages: string[], options: OptionValues) {
const [name, version] = pkg.split(/[#@]/, 2);
await loader.loadPackage(name, version);
}

if (options.export) {
const fplExport = await loader.exportDB();
if (fplExport.mimeType === 'application/x-sqlite3') {
const exportPath = path.join(process.cwd(), 'FPL.sqlite');
fs.writeFileSync(exportPath, fplExport.data);
logger.info(`Exported FPL database to ${exportPath}`);
}
}
}

async function app() {
Expand All @@ -70,6 +79,7 @@ async function app() {
'where to save packages to and load definitions from (default is the local FHIR cache)'
)
.option('-d, --debug', 'output extra debugging information')
.option('-e, --export', 'export a SQLite DB file with data from the loaded packages')
.action(install);

await program.parseAsync(process.argv);
Expand Down
144 changes: 26 additions & 118 deletions src/cache/DiskBasedPackageCache.ts
View file
Original file line number Diff line number Diff line change
@@ -1,35 +1,23 @@
import path from 'path';
import fs from 'fs-extra';
import { LogFunction } from '../utils';
import {
PackageCache,
PackageCacheOptions,
LOCAL_PACKAGE_NAME,
LOCAL_PACKAGE_VERSION
} from './PackageCache';
import { PackageCache, PackageCacheOptions } from './PackageCache';
import temp from 'temp';
import * as tar from 'tar';
import { Readable } from 'stream';
import { pipeline } from 'stream/promises';
import { LRUCache } from 'mnemonist';
import { Fhir as FHIRConverter } from 'fhir/fhir';

export class DiskBasedPackageCache implements PackageCache {
private log: LogFunction;
private localResourceFolders: string[];
private fhirConverter: FHIRConverter;
private lruCache: LRUCache<string, any>;

constructor(
private cachePath: string,
localResourceFolders: string[] = [],
options: PackageCacheOptions = {}
) {
this.log = options.log ?? (() => {});
this.localResourceFolders = localResourceFolders.map(f => path.resolve(f));
this.fhirConverter = new FHIRConverter();
// TODO: Make Cache Size Configurable
this.lruCache = new LRUCache<string, any>(500);
}

async cachePackageTarball(name: string, version: string, data: Readable): Promise<string> {
Expand All @@ -51,32 +39,19 @@ export class DiskBasedPackageCache implements PackageCache {
}

isPackageInCache(name: string, version: string): boolean {
if (isLocalPackage(name, version)) {
return true;
}
return fs.existsSync(path.resolve(this.cachePath, `${name}#${version}`));
}

getPackagePath(name: string, version: string): string | undefined {
if (this.isPackageInCache(name, version)) {
if (isLocalPackage(name, version)) {
return this.localResourceFolders.join(';');
}
return path.resolve(this.cachePath, `${name}#${version}`);
}
}

getPackageJSONPath(name: string, version: string): string | undefined {
if (!isLocalPackage(name, version)) {
const jsonPath = path.resolve(
this.cachePath,
`${name}#${version}`,
'package',
'package.json'
);
if (fs.existsSync(jsonPath)) {
return jsonPath;
}
const jsonPath = path.resolve(this.cachePath, `${name}#${version}`, 'package', 'package.json');
if (fs.existsSync(jsonPath)) {
return jsonPath;
}
}

Expand All @@ -85,104 +60,37 @@ export class DiskBasedPackageCache implements PackageCache {
return [];
}

if (isLocalPackage(name, version)) {
const spreadSheetCounts = new Map<string, number>();
const invalidFileCounts = new Map<string, number>();
const resourcePaths: string[] = [];
this.localResourceFolders.forEach(folder => {
let spreadSheetCount = 0;
let invalidFileCount = 0;
fs.readdirSync(folder, { withFileTypes: true })
.filter(entry => {
if (!entry.isFile()) {
return false;
} else if (/\.json$/i.test(entry.name)) {
return true;
} else if (/-spreadsheet.xml/i.test(entry.name)) {
spreadSheetCount++;
this.log(
'debug',
`Skipped spreadsheet XML file: ${path.resolve(entry.path, entry.name)}`
);
return false;
} else if (/\.xml/i.test(entry.name)) {
const xml = fs.readFileSync(path.resolve(entry.path, entry.name)).toString();
if (/<\?mso-application progid="Excel\.Sheet"\?>/m.test(xml)) {
spreadSheetCount++;
this.log(
'debug',
`Skipped spreadsheet XML file: ${path.resolve(entry.path, entry.name)}`
);
return false;
}
return true;
}
invalidFileCount++;
this.log(
'debug',
`Skipped non-JSON / non-XML file: ${path.resolve(entry.path, entry.name)}`
);
return false;
})
.forEach(entry => resourcePaths.push(path.resolve(entry.path, entry.name)));
spreadSheetCounts.set(folder, spreadSheetCount);
invalidFileCounts.set(folder, invalidFileCount);
});
spreadSheetCounts.forEach((count, folder) => {
if (count) {
this.log(
'info',
`Found ${count} spreadsheet(s) in directory: ${folder}. SUSHI does not support spreadsheets, so any resources in the spreadsheets will be ignored. To see the skipped files in the logs, run SUSHI with the "--log-level debug" flag.`
);
}
});
invalidFileCounts.forEach((count, folder) => {
if (count) {
this.log(
'info',
`Found ${count} non-JSON / non-XML file(s) in directory: ${folder}. SUSHI only processes resource files with JSON or XML extensions. To see the skipped files in the logs, run SUSHI with the "--log-level debug" flag.`
);
}
});
return resourcePaths;
} else {
const contentPath = path.resolve(this.cachePath, `${name}#${version}`, 'package');
return fs
.readdirSync(contentPath, { withFileTypes: true })
.filter(entry => entry.isFile() && /^[^.].*\.json$/i.test(entry.name))
.map(entry => path.resolve(entry.path, entry.name));
}
const contentPath = path.resolve(this.cachePath, `${name}#${version}`, 'package');
// Since every OS may load paths in a different order, ensure consistency by sorting the final paths
return fs
.readdirSync(contentPath, { withFileTypes: true })
.filter(entry => entry.isFile() && /^[^.].*\.json$/i.test(entry.name))
.map(entry => path.resolve(entry.path, entry.name))
.sort();
}

getResourceAtPath(resourcePath: string) {
let resource = this.lruCache.get(resourcePath);
if (!resource) {
if (/.xml$/i.test(resourcePath)) {
try {
const xml = fs.readFileSync(resourcePath).toString();
resource = this.fhirConverter.xmlToObj(xml);
} catch {
throw new Error(`Failed to get XML resource at path ${resourcePath}`);
}
} else if (/.json$/i.test(resourcePath)) {
try {
resource = fs.readJSONSync(resourcePath);
} catch {
throw new Error(`Failed to get JSON resource at path ${resourcePath}`);
}
} else {
throw new Error(`Failed to find XML or JSON file at path ${resourcePath}`);
let resource;
if (/.xml$/i.test(resourcePath)) {
try {
const xml = fs.readFileSync(resourcePath).toString();
resource = this.fhirConverter.xmlToObj(xml);
} catch {
throw new Error(`Failed to get XML resource at path ${resourcePath}`);
}
} else if (/.json$/i.test(resourcePath)) {
try {
resource = fs.readJSONSync(resourcePath);
} catch {
throw new Error(`Failed to get JSON resource at path ${resourcePath}`);
}
this.lruCache.set(resourcePath, resource);
} else {
throw new Error(`Failed to find XML or JSON file at path ${resourcePath}`);
}
return resource;
}
}

function isLocalPackage(name: string, version: string) {
return name === LOCAL_PACKAGE_NAME && version === LOCAL_PACKAGE_VERSION;
}

/**
* This function takes a package which contains contents at the same level as the "package" folder, and nests
* all that content within the "package" folder.
Expand Down
3 changes: 0 additions & 3 deletions src/cache/PackageCache.ts
View file
Original file line number Diff line number Diff line change
@@ -1,9 +1,6 @@
import { Readable } from 'stream';
import { LogFunction } from '../utils';

export const LOCAL_PACKAGE_NAME = 'LOCAL';
export const LOCAL_PACKAGE_VERSION = 'LOCAL';

export type PackageCacheOptions = {
log?: LogFunction;
};
Expand Down
2 changes: 1 addition & 1 deletion src/current/BuildDotFhirDotOrgClient.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ export class BuildDotFhirDotOrgClient implements CurrentBuildClient {
this.log = options.log ?? (() => {});
}

async downloadCurrentBuild(name: string, branch: string | null): Promise<Readable> {
async downloadCurrentBuild(name: string, branch?: string): Promise<Readable> {
const version = branch ? `current$${branch}` : 'current';
const baseURL = await this.getCurrentBuildBaseURL(name, branch);
if (!baseURL) {
Expand Down
2 changes: 1 addition & 1 deletion src/current/CurrentBuildClient.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,6 @@ export type CurrentBuildClientOptions = {
};

export interface CurrentBuildClient {
downloadCurrentBuild(name: string, branch: string | null): Promise<Readable>;
downloadCurrentBuild(name: string, branch?: string): Promise<Readable>;
getCurrentBuildDate(name: string, branch?: string): Promise<string>;
}
1 change: 1 addition & 0 deletions src/db/PackageDB.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,5 @@ export interface PackageDB {
findResourceInfos(key: string, options?: FindResourceInfoOptions): ResourceInfo[];
findResourceInfo(key: string, options?: FindResourceInfoOptions): ResourceInfo | undefined;
getPackageStats(name: string, version: string): PackageStats | undefined;
exportDB(): Promise<{ mimeType: string; data: Buffer }>;
}
Loading

0 comments on commit ab29c50

Please sign in to comment.