Virtual Packages and SUSHI Compatibility Changes

Various changes to better support the needs of SUSHI while remaining neutral for other environments and applications. Including:
* VirtualPackage interface for pseudo packages not in the registry or local FHIR cache
* DiskBased and InMemory implementations of Virtual Package
* Support for loading VirtualPackages in the BasePackageLoader
* Support for '|version' when searching by key
* New exportDB function in SQLJSPackageDB and BasePackageLoader for debugging purposes
* Added sort to find functions' options, with initial ByLoadOrder and ByType implementations (the original type argument is now just a filter, no longer affecting sort)
* Added resolveVersion function to RegistryClient interface and corresponding implementations
* Moved LRU cache from DiskBasedPackageCache to BasePackageLoader

README.md

@@ -78,6 +78,7 @@ FHIR Package Loader can be used as a library to download FHIR packages, query th
 ```ts
 export interface PackageLoader {
   loadPackage(name: string, version: string): Promise<LoadStatus>;
+  loadVirtualPackage(pkg: VirtualPackage): Promise<LoadStatus>;
   getPackageLoadStatus(name: string, version: string): LoadStatus;
   findPackageInfos(name: string): PackageInfo[];
   findPackageInfo(name: string, version: string): PackageInfo | undefined;
@@ -101,6 +102,7 @@ The [default PackageLoader](src/loader/DefaultPackageLoader.ts) implementation p
 * the standard FHIR registry is used (`packages.fhir.org`) for downloading published packages, falling back to `packages2.fhir.org` when necessary
   * unless an `FPL_REGISTRY` environment variable is defined, in which case its value is used as the URL for an NPM registry to use _instead_ of the standard FHIR registry
 * the `build.fhir.org` build server is used for downloading _current_ builds of packages
+* a 500-item LRU in-memory cache is used to minimize repeated disk reads for resource files
 
 To instantiate the default `PackageLoader`, import the asynchronous `defaultPackageLoader` function and invoke it, optionally passing in an `options` object with a log method to use for logging:
 
@@ -116,9 +118,7 @@ if (status !== LoadStatus.LOADED) {
 }
 ```
 
-To instantiate the default `PackageLoader` with a set of standalone JSON or XML resources that should be pre-loaded, use the `defaultPackageLoaderWithLocalResources` function instead, passing in an array of file paths to folders containing the resources to load.
-
-For more control over the `PackageLoader`, use the [BasePackageLoader](src/loader/BasePackageLoader.ts). This allows you to specify the [PackageDB](src/db), [PackageCache](src/cache), [RegistryClient](src/registry), and [CurrentBuildClient](src/current) you wish to use. FHIRPackageLoader comes with implementations of each of these, but you may also provide your own implementations that adhere to the relevant interfaces.
+For more control over the `PackageLoader`, use the [BasePackageLoader](src/loader/BasePackageLoader.ts). This allows you to specify the [PackageDB](src/db), [PackageCache](src/cache), [RegistryClient](src/registry), and [CurrentBuildClient](src/current) you wish to use FHIRPackageLoader comes with implementations of each of these, but you may also provide your own implementations that adhere to the relevant interfaces. The BasePackageLoader also allows you to configure the size of the in-memory LRU resource cache.
 
 ### PackageLoader Functions
 
@@ -128,6 +128,10 @@ The `PackageLoader` interface provides the following functions:
 
 Loads the specified package version. The version may be a specific version (e.g., `1.2.3`), a wildcard patch version (e.g., `1.2.x`), `dev` (to indicate the local development build in your FHIR cache), `current` (to indicate the current master/main build), `current$branchname` (to indicate the current build on a specific branch), or `latest` (to indicate the most recent published version). Returns the [LoadStatus](src/loader/PackageLoader.ts).
 
+#### `loadVirtualPackage(pkg: VirtualPackage): Promise<LoadStatus>`
+
+Loads a resources from a passed in implementation of the [VirtualPackage](src/virtual/VirtualPackage.ts) interface. This allows for "virtual" packages that do not come from a registry nor are stored in the local FHIR package cache. The [DiskBasedVirtualPackage](src/virtual/DiskBasedVirtualPackage.ts) implementation allows resources from arbitrary file paths (folders or direct files) to be loaded as a package. The [InMemoryVirtualPackage](src/virtual/InMemoryVirtualPackage.ts) implementation allows resources in a Map to be loaded as a package. Developers may also provide their own implementation of the VirtualPackage interface. Returns the [LoadStatus](src/loader/PackageLoader.ts).
+
 #### `getPackageLoadStatus(name: string, version: string): LoadStatus`
 
 Gets the [LoadStatus](src/loader/PackageLoader.ts) for the specified package version. The returned value will be `LoadStatus.LOADED` if it is already loaded, `LoadStatus.NOT_LOADED` if it has not yet been loaded, or `LoadStatus.FAILED` if it was attempted but failed to load. This function supports specific versions (e.g. `1.2.3`), `dev`, `current`, and `current$branchname`. It does _not_ support wildcard patch versions (e.g., `1.2.x`) nor does it support the `latest` version.

src/app.ts

@@ -37,7 +37,7 @@ async function install(fhirPackages: string[], options: OptionValues) {
   const SQL = await initSqlJs();
   const packageDB = new SQLJSPackageDB(new SQL.Database());
   const fhirCache = options.cachePath ?? path.join(os.homedir(), '.fhir', 'packages');
-  const packageCache = new DiskBasedPackageCache(fhirCache, [], { log });
+  const packageCache = new DiskBasedPackageCache(fhirCache, { log });
   const registryClient = new DefaultRegistryClient({ log });
   const buildClient = new BuildDotFhirDotOrgClient({ log });
   const loader = new BasePackageLoader(packageDB, packageCache, registryClient, buildClient, {

src/cache/DiskBasedPackageCache.ts

@@ -1,35 +1,23 @@
 import path from 'path';
 import fs from 'fs-extra';
 import { LogFunction } from '../utils';
-import {
-  PackageCache,
-  PackageCacheOptions,
-  LOCAL_PACKAGE_NAME,
-  LOCAL_PACKAGE_VERSION
-} from './PackageCache';
+import { PackageCache, PackageCacheOptions } from './PackageCache';
 import temp from 'temp';
 import * as tar from 'tar';
 import { Readable } from 'stream';
 import { pipeline } from 'stream/promises';
-import { LRUCache } from 'mnemonist';
 import { Fhir as FHIRConverter } from 'fhir/fhir';
 
 export class DiskBasedPackageCache implements PackageCache {
   private log: LogFunction;
-  private localResourceFolders: string[];
   private fhirConverter: FHIRConverter;
-  private lruCache: LRUCache<string, any>;
 
   constructor(
     private cachePath: string,
-    localResourceFolders: string[] = [],
     options: PackageCacheOptions = {}
   ) {
     this.log = options.log ?? (() => {});
-    this.localResourceFolders = localResourceFolders.map(f => path.resolve(f));
     this.fhirConverter = new FHIRConverter();
-    // TODO: Make Cache Size Configurable
-    this.lruCache = new LRUCache<string, any>(500);
   }
 
   async cachePackageTarball(name: string, version: string, data: Readable): Promise<string> {
@@ -51,32 +39,19 @@ export class DiskBasedPackageCache implements PackageCache {
   }
 
   isPackageInCache(name: string, version: string): boolean {
-    if (isLocalPackage(name, version)) {
-      return true;
-    }
     return fs.existsSync(path.resolve(this.cachePath, `${name}#${version}`));
   }
 
   getPackagePath(name: string, version: string): string | undefined {
     if (this.isPackageInCache(name, version)) {
-      if (isLocalPackage(name, version)) {
-        return this.localResourceFolders.join(';');
-      }
       return path.resolve(this.cachePath, `${name}#${version}`);
     }
   }
 
   getPackageJSONPath(name: string, version: string): string | undefined {
-    if (!isLocalPackage(name, version)) {
-      const jsonPath = path.resolve(
-        this.cachePath,
-        `${name}#${version}`,
-        'package',
-        'package.json'
-      );
-      if (fs.existsSync(jsonPath)) {
-        return jsonPath;
-      }
+    const jsonPath = path.resolve(this.cachePath, `${name}#${version}`, 'package', 'package.json');
+    if (fs.existsSync(jsonPath)) {
+      return jsonPath;
     }
   }
 
@@ -85,104 +60,37 @@ export class DiskBasedPackageCache implements PackageCache {
       return [];
     }
 
-    if (isLocalPackage(name, version)) {
-      const spreadSheetCounts = new Map<string, number>();
-      const invalidFileCounts = new Map<string, number>();
-      const resourcePaths: string[] = [];
-      this.localResourceFolders.forEach(folder => {
-        let spreadSheetCount = 0;
-        let invalidFileCount = 0;
-        fs.readdirSync(folder, { withFileTypes: true })
-          .filter(entry => {
-            if (!entry.isFile()) {
-              return false;
-            } else if (/\.json$/i.test(entry.name)) {
-              return true;
-            } else if (/-spreadsheet.xml/i.test(entry.name)) {
-              spreadSheetCount++;
-              this.log(
-                'debug',
-                `Skipped spreadsheet XML file: ${path.resolve(entry.path, entry.name)}`
-              );
-              return false;
-            } else if (/\.xml/i.test(entry.name)) {
-              const xml = fs.readFileSync(path.resolve(entry.path, entry.name)).toString();
-              if (/<\?mso-application progid="Excel\.Sheet"\?>/m.test(xml)) {
-                spreadSheetCount++;
-                this.log(
-                  'debug',
-                  `Skipped spreadsheet XML file: ${path.resolve(entry.path, entry.name)}`
-                );
-                return false;
-              }
-              return true;
-            }
-            invalidFileCount++;
-            this.log(
-              'debug',
-              `Skipped non-JSON / non-XML file: ${path.resolve(entry.path, entry.name)}`
-            );
-            return false;
-          })
-          .forEach(entry => resourcePaths.push(path.resolve(entry.path, entry.name)));
-        spreadSheetCounts.set(folder, spreadSheetCount);
-        invalidFileCounts.set(folder, invalidFileCount);
-      });
-      spreadSheetCounts.forEach((count, folder) => {
-        if (count) {
-          this.log(
-            'info',
-            `Found ${count} spreadsheet(s) in directory: ${folder}. SUSHI does not support spreadsheets, so any resources in the spreadsheets will be ignored. To see the skipped files in the logs, run SUSHI with the "--log-level debug" flag.`
-          );
-        }
-      });
-      invalidFileCounts.forEach((count, folder) => {
-        if (count) {
-          this.log(
-            'info',
-            `Found ${count} non-JSON / non-XML file(s) in directory: ${folder}. SUSHI only processes resource files with JSON or XML extensions. To see the skipped files in the logs, run SUSHI with the "--log-level debug" flag.`
-          );
-        }
-      });
-      return resourcePaths;
-    } else {
-      const contentPath = path.resolve(this.cachePath, `${name}#${version}`, 'package');
-      return fs
-        .readdirSync(contentPath, { withFileTypes: true })
-        .filter(entry => entry.isFile() && /^[^.].*\.json$/i.test(entry.name))
-        .map(entry => path.resolve(entry.path, entry.name));
-    }
+    const contentPath = path.resolve(this.cachePath, `${name}#${version}`, 'package');
+    // Since every OS may load paths in a different order, ensure consistency by sorting the final paths
+    return fs
+      .readdirSync(contentPath, { withFileTypes: true })
+      .filter(entry => entry.isFile() && /^[^.].*\.json$/i.test(entry.name))
+      .map(entry => path.resolve(entry.path, entry.name))
+      .sort();
   }
 
   getResourceAtPath(resourcePath: string) {
-    let resource = this.lruCache.get(resourcePath);
-    if (!resource) {
-      if (/.xml$/i.test(resourcePath)) {
-        try {
-          const xml = fs.readFileSync(resourcePath).toString();
-          resource = this.fhirConverter.xmlToObj(xml);
-        } catch {
-          throw new Error(`Failed to get XML resource at path ${resourcePath}`);
-        }
-      } else if (/.json$/i.test(resourcePath)) {
-        try {
-          resource = fs.readJSONSync(resourcePath);
-        } catch {
-          throw new Error(`Failed to get JSON resource at path ${resourcePath}`);
-        }
-      } else {
-        throw new Error(`Failed to find XML or JSON file at path ${resourcePath}`);
+    let resource;
+    if (/.xml$/i.test(resourcePath)) {
+      try {
+        const xml = fs.readFileSync(resourcePath).toString();
+        resource = this.fhirConverter.xmlToObj(xml);
+      } catch {
+        throw new Error(`Failed to get XML resource at path ${resourcePath}`);
+      }
+    } else if (/.json$/i.test(resourcePath)) {
+      try {
+        resource = fs.readJSONSync(resourcePath);
+      } catch {
+        throw new Error(`Failed to get JSON resource at path ${resourcePath}`);
       }
-      this.lruCache.set(resourcePath, resource);
+    } else {
+      throw new Error(`Failed to find XML or JSON file at path ${resourcePath}`);
     }
     return resource;
   }
 }
 
-function isLocalPackage(name: string, version: string) {
-  return name === LOCAL_PACKAGE_NAME && version === LOCAL_PACKAGE_VERSION;
-}
-
 /**
  * This function takes a package which contains contents at the same level as the "package" folder, and nests
  * all that content within the "package" folder.

src/cache/PackageCache.ts

@@ -1,9 +1,6 @@
 import { Readable } from 'stream';
 import { LogFunction } from '../utils';
 
-export const LOCAL_PACKAGE_NAME = 'LOCAL';
-export const LOCAL_PACKAGE_VERSION = 'LOCAL';
-
 export type PackageCacheOptions = {
   log?: LogFunction;
 };

src/current/BuildDotFhirDotOrgClient.ts

@@ -9,7 +9,7 @@ export class BuildDotFhirDotOrgClient implements CurrentBuildClient {
     this.log = options.log ?? (() => {});
   }
 
-  async downloadCurrentBuild(name: string, branch: string | null): Promise<Readable> {
+  async downloadCurrentBuild(name: string, branch?: string): Promise<Readable> {
     const version = branch ? `current$${branch}` : 'current';
     const baseURL = await this.getCurrentBuildBaseURL(name, branch);
     if (!baseURL) {

src/current/CurrentBuildClient.ts

@@ -6,6 +6,6 @@ export type CurrentBuildClientOptions = {
 };
 
 export interface CurrentBuildClient {
-  downloadCurrentBuild(name: string, branch: string | null): Promise<Readable>;
+  downloadCurrentBuild(name: string, branch?: string): Promise<Readable>;
   getCurrentBuildDate(name: string, branch?: string): Promise<string>;
 }

src/db/PackageDB.ts

@@ -9,4 +9,5 @@ export interface PackageDB {
   findResourceInfos(key: string, options?: FindResourceInfoOptions): ResourceInfo[];
   findResourceInfo(key: string, options?: FindResourceInfoOptions): ResourceInfo | undefined;
   getPackageStats(name: string, version: string): PackageStats | undefined;
+  exportDB(): Promise<{ mimeType: string; data: Buffer }>;
 }

src/db/SQLJSPackageDB.ts

@@ -100,6 +100,8 @@ const INSERT_RESOURCE = `INSERT INTO resource
     :resourcePath
   );`;
 
+const SD_FLAVORS = ['Extension', 'Logical', 'Profile', 'Resource', 'Type'];
+
 export class SQLJSPackageDB implements PackageDB {
   private insertPackageStmt: Statement;
   private insertResourceStmt: Statement;
@@ -233,16 +235,20 @@ export class SQLJSPackageDB implements PackageDB {
       key = '';
     }
     // TODO: Upgrade to class-level property once query and parameters are sorted out
-    // TODO: Support versions when canonical form is used
+    const [keyText, ...keyVersion] = key.split('|');
     const bindStmt: { [key: string]: string } = {};
     let findStmt = 'SELECT * FROM resource WHERE ';
-    if (key !== '*') {
+    if (keyText !== '*') {
       // special case for selecting all
-      bindStmt[':key'] = key;
+      bindStmt[':key'] = keyText;
       findStmt += '(id = :key OR name = :key OR url = :key)';
     } else {
       findStmt += '1';
     }
+    if (keyVersion.length) {
+      bindStmt[':version'] = keyVersion.join('|');
+      findStmt += ' AND version = :version';
+    }
     if (options.scope?.length) {
       const [packageName, ...packageVersion] = options.scope.split('|');
       bindStmt[':packageName'] = packageName;
@@ -255,11 +261,29 @@ export class SQLJSPackageDB implements PackageDB {
     if (options.type?.length) {
       const conditions = options.type.map((t, i) => {
         bindStmt[`:type${i}`] = t;
-        return `(sdFlavor = :type${i} OR resourceType = :type${i})`;
+        const field = SD_FLAVORS.includes(t) ? 'sdFlavor' : 'resourceType';
+        return `${field} = :type${i}`;
       });
-      findStmt += ` AND (${conditions.join(' OR ')}) ORDER BY ${conditions.map(c => `${c} DESC`).join(', ')}, rowid DESC`;
-    } else {
-      findStmt += ' ORDER BY rowid DESC';
+      findStmt += ` AND (${conditions.join(' OR ')})`;
+    }
+    if (options.sort) {
+      const sortExpressions: string[] = [];
+      options.sort.forEach(s => {
+        switch (s.sortBy) {
+          case 'LoadOrder':
+            sortExpressions.push(`rowId ${s.ascending ? 'ASC' : 'DESC'}`);
+            break;
+          case 'Type':
+            (s.types as string[]).forEach((t, i) => {
+              bindStmt[`:sortType${i}`] = t;
+              const field = SD_FLAVORS.includes(t) ? 'sdFlavor' : 'resourceType';
+              // This sort expression is weird, but... it's the only way it works as expected!
+              sortExpressions.push(`(${field} = :sortType${i} OR NULL) DESC`);
+            });
+            break;
+        }
+      });
+      findStmt += ` ORDER BY ${sortExpressions.join(', ')}`;
     }
     if (options.limit) {
       bindStmt[':limit'] = String(options.limit);
@@ -311,6 +335,11 @@ export class SQLJSPackageDB implements PackageDB {
     };
   }
 
+  exportDB(): Promise<{ mimeType: string; data: Buffer }> {
+    const data = this.db.export();
+    return Promise.resolve({ mimeType: 'application/x-sqlite3', data: Buffer.from(data) });
+  }
+
   logPackageTable() {
     const res = this.db.exec('SELECT * FROM package');
     console.log(util.inspect(res, false, 3, true));

src/errors/LatestVersionUnavailableError.ts

@@ -1,15 +1,12 @@
 export class LatestVersionUnavailableError extends Error {
   constructor(
     public packageName: string,
-    public customRegistry?: string,
     public isPatchWildCard?: boolean
   ) {
     super(
       `Latest ${
         isPatchWildCard ? 'patch ' : ''
-      }version of package ${packageName} could not be determined from the ${
-        customRegistry ? 'custom ' : ''
-      }FHIR package registry${customRegistry ? ` ${customRegistry}` : ''}`
+      }version of package ${packageName} could not be determined from the package registry`
     );
   }
 }

src/index.ts

@@ -5,4 +5,6 @@ export * from './current';
 export * from './db';
 export * from './loader';
 export * from './package';
+export * from './sort';
 export * from './registry';
+export * from './virtual';