Dynamic imports (#40)

* rebased onto master * add examples for dynamic import * WIP, at least one test isn't working * tests pass * all tests pass * test++ * test++ * test++ * timerManager restore --------- Co-authored-by: Geoffrey Hendrey <[email protected]>
cisco-open · Jan 12, 2024 · 14bead3 · 14bead3
1 parent 58da28f
commit 14bead3
Show file tree

Hide file tree

Showing 12 changed files with 260 additions and 114 deletions.
diff --git a/README.md b/README.md
@@ -811,12 +811,36 @@ Individual JSONata programs are embedded in JSON files between `${..}`. What is
 The input, by default, is the object or array that the expression resides in. For instance in the example **above**, you can see that the JSONata `$` variable refers to the array itself. Therefore, expressions like `$[0]`
 refer to the first element of the array. 
 ## Rerooting Expressions
-In the example below, we want `player1` to refer to `/player1` (the field named 'player1' at the root of the document). 
-But our expression `greeting & ', ' &  player1` is located deep in the document at `/dialog/part1`. So how can we cause 
-the root of the document to be the input to the JSONata expression `greeting & ', ' &  player1`? 
+In Stated templates, one way to declare a JSONata expression is by surrounding it by "dollars moustaches".
+E.g `${...some expression...}`. JSONata expressions always have a [context](https://docs.jsonata.org/programming#built-in-variables).
+The `$` variable always points to the current context. The `$$` variable always points to the input (root context) for an 
+expression.
+In a Stated template, the root context for an expression is the object in which the expression is contained. For
+Example:
+```json
+> .init -f "example/context.json"
+{
+   "a": {
+      "b": "${[c,' and ',$.c,' and ',$$.c,' are the same thing. $ (current context) is /a, the object in which this expression resides']~>$join}",
+      "c": "hello"
+   }
+}
+> .out
+{
+   "a": {
+      "b": "hello and hello and hello are the same thing. $ (current context) is /a, the object in which this expression resides",
+      "c": "hello"
+   }
+}
+```
+Now we will show how we can change the context of an expression using 'rerooting.' Rerooting allows the expression's root
+context to be pointed anywhere in the json document.
+In the example below, consider `greeting & ', ' &  player1'`. We want `player1` to refer to the content at json pointer `/player1` (the field named 'player1' at the root of the document). 
+But our expression `greeting & ', ' &  player1` is located deep in the document at `/dialog/partI`. So how can we cause 
+the root of the document to be the context for the JSONata expression `greeting & ', ' &  player1`? 
 You can reroot an expression in a different part of the document using relative rooting `../${<expr>}` syntax or you can root an
 at the absolute doc root with `/${<expr>}`. The example below shows how expressions located below the root object, can
-explicitly set their input using the rooting syntax. Both absolute rooting, `/${...}` and relative rooting `../${...}`
+explicitly set their context using the rooting syntax. Both absolute rooting, `/${...}` and relative rooting `../${...}`
 are shown.
 
 ```json
@@ -852,8 +876,37 @@ are shown.
     }
   }
 }
+```
+An advanced rerooting operator is the `//` absolute root operator. The `/` rooting operator, that we showed above,  will never allow the expression
+to 'escape' outside of the template it was defined in. But what if we intend for a template to be imported into another template
+and we expect there to be a variable defined in the other template that we should use? This is where the `//` absolute root
+operator can be used. The `//` operator will set the expression context to the absolute root of whatever the final document is
+after all imports have been performed.
+```json
+> .init -f "example/absoluteRoot.json"
+{
+   "to": "!${'Professor Falken'}",
+   "greeting": "//${'Hello, ' & to}"
+}
+> .out
+{
+   "greeting": "Hello, Professor Falken"
+}
+> .init -f "example/importsAbsoluteRoot.json"
+{
+   "to": "Joshua",
+   "message": "${$import('example/absoluteRoot.json')}"
+}
+> .out
+{
+   "to": "Joshua",
+   "message": {
+      "greeting": "Hello, Joshua"
+   }
+}
 
 ```
+
 ## DAG
 Templates can grow complex, and embedded expressions have dependencies on both literal fields and other calculated
 expressions. stated is at its core a data flow engine. Stated analyzes the abstract syntax tree (AST) of JSONata 
@@ -1244,41 +1297,41 @@ remote templates (or local literal templates) into the current template
 }
 > .note "Now let's use the import function on the template"
 "============================================================="
-> .init -f "example/ex16.json"
+> .init -f example/ex16.json
 {
-  "noradCommander": "${ norad.commanderDetails  }",
-  "norad": "${ $fetch('https://raw.githubusercontent.com/geoffhendrey/jsonataplay/main/norad.json') ~> handleRes ~> $import('/norad')}",
-  "handleRes": "${ function($res){$res.ok? $res.json():{'error': $res.status}} }"
+   "noradCommander": "${ norad.commanderDetails  }",
+   "norad": "${ $fetch('https://raw.githubusercontent.com/geoffhendrey/jsonataplay/main/norad.json') ~> handleRes ~> $import}",
+   "handleRes": "${ function($res){$res.ok? $res.json():{'error': $res.status}} }"
 }
 > .out
 {
-  "noradCommander": {
-    "fullName": "Jack Beringer",
-    "salutation": "General Jack Beringer",
-    "systemsUnderCommand": 4
-  },
-  "norad": {
-    "commanderDetails": {
+   "noradCommander": {
       "fullName": "Jack Beringer",
       "salutation": "General Jack Beringer",
       "systemsUnderCommand": 4
-    },
-    "organization": "NORAD",
-    "location": "Cheyenne Mountain Complex, Colorado",
-    "commander": {
-      "firstName": "Jack",
-      "lastName": "Beringer",
-      "rank": "General"
-    },
-    "purpose": "Provide aerospace warning, air sovereignty, and defense for North America",
-    "systems": [
-      "Ballistic Missile Early Warning System (BMEWS)",
-      "North Warning System (NWS)",
-      "Space-Based Infrared System (SBIRS)",
-      "Cheyenne Mountain Complex"
-    ]
-  },
-  "handleRes": "{function:}"
+   },
+   "norad": {
+      "commanderDetails": {
+         "fullName": "Jack Beringer",
+         "salutation": "General Jack Beringer",
+         "systemsUnderCommand": 4
+      },
+      "organization": "NORAD",
+      "location": "Cheyenne Mountain Complex, Colorado",
+      "commander": {
+         "firstName": "Jack",
+         "lastName": "Beringer",
+         "rank": "General"
+      },
+      "purpose": "Provide aerospace warning, air sovereignty, and defense for North America",
+      "systems": [
+         "Ballistic Missile Early Warning System (BMEWS)",
+         "North Warning System (NWS)",
+         "Space-Based Infrared System (SBIRS)",
+         "Cheyenne Mountain Complex"
+      ]
+   },
+   "handleRes": "{function:}"
 }
 > .note "You can see above that 'import' makes it behave as a template, not raw JSON."
 "============================================================="

diff --git a/example/absoluteRoot.json b/example/absoluteRoot.json
@@ -0,0 +1,4 @@
+{
+  "to": "!${'Professor Falken'}",
+  "greeting": "//${'Hello, ' & to}"
+}
diff --git a/example/context.json b/example/context.json
@@ -0,0 +1,6 @@
+{
+  "a": {
+    "b": "${[c,' and ',$.c,' and ',$$.c,' are the same thing. $ (current context) is /a, the object in which this expression resides']~>$join}",
+    "c": "hello"
+  }
+}
diff --git a/example/ex16.json b/example/ex16.json
@@ -1,5 +1,5 @@
 {
   "noradCommander": "${ norad.commanderDetails  }",
-  "norad": "${ $fetch('https://raw.githubusercontent.com/geoffhendrey/jsonataplay/main/norad.json') ~> handleRes ~> $import('/norad')}",
+  "norad": "${ $fetch('https://raw.githubusercontent.com/geoffhendrey/jsonataplay/main/norad.json') ~> handleRes ~> $import}",
   "handleRes": "${ function($res){$res.ok? $res.json():{'error': $res.status}} }"
 }
diff --git a/example/experimental/main.yaml b/example/experimental/main.yaml
@@ -0,0 +1,6 @@
+name: Main
+SAY_HELLO: "${$fetch('https://raw.githubusercontent.com/geoffhendrey/jsonataplay/main/sayhello.json').json()}"
+view:
+  - "/${SAY_HELLO ~> |props|{'name':'world'}| ~> $import}"
+  - "/${SAY_HELLO ~> |props|{'name':'Universe'}| ~> $import}"
+  - "/${SAY_HELLO ~> |props|{'name':'Galaxy'}| ~> $import}"
diff --git a/example/experimental/main2.yaml b/example/experimental/main2.yaml
@@ -0,0 +1,12 @@
+name: Main
+SAY_HELLO: "${$fetch('https://raw.githubusercontent.com/geoffhendrey/jsonataplay/main/sayhello.json').json()}"
+now: "${ function(){$set('/timeString', $Date())} }"
+tick: "${ $setInterval(now , 1000) }"
+timeString: ''
+worldTime: ${ "World! " & timeString }
+universeTime: ${ "Universe! " & timeString }
+galaxyTime: ${ "Galaxy! " & timeString }
+view:
+  - "../${SAY_HELLO ~> |props|{'name':'${$$.worldTime}'}| ~> $import}"
+  - "../${SAY_HELLO ~> |props|{'name':'${$$.universeTime}'}| ~> $import}"
+  - "../${SAY_HELLO ~> |props|{'name':'${$$.galaxyTime}'}| ~> $import}"
diff --git a/example/importsAbsoluteRoot.json b/example/importsAbsoluteRoot.json
@@ -0,0 +1,4 @@
+{
+  "to": "Joshua",
+  "message": "${$import('example/absoluteRoot.json')}"
+}
diff --git a/src/CliCore.ts b/src/CliCore.ts
@@ -114,7 +114,7 @@ export default class CliCore {
             this.templateProcessor.close();
         }
         const parsed = CliCore.parseInitArgs(replCmdInputStr);
-        const {filepath, tags,oneshot, options, xf:contextFilePath, importPath, tail} = parsed;
+        const {filepath, tags,oneshot, options, xf:contextFilePath, importPath=this.currentDirectory, tail} = parsed;
         if(filepath===undefined){
             return undefined;
         }
@@ -400,66 +400,65 @@ export default class CliCore {
 
 
     public async open(directory: string = this.currentDirectory) {
-    if(directory === ""){
-        directory = this.currentDirectory;
-    }
+        if(directory === ""){
+            directory = this.currentDirectory;
+        }
 
-    let files: string[] = undefined;
-    try {
-        // Read all files from the directory
-        files = await fs.promises.readdir(directory);
-    } catch (error) {
-        console.log(`Error reading directory ${directory}: ${error}`);
-        console.log('Changed directory with .cd or .open an/existing/directory');
-        this.replServer.displayPrompt();
-        return {error: `Error reading directory ${directory}: ${error}`};
-    }
-    // Filter out only .json and .yaml files
-    const templateFiles: string[] = files.filter(file => file.endsWith('.json') || file.endsWith('.yaml'));
-
-    // Display the list of files to the user
-    templateFiles.forEach((file, index) => {
-        console.log(`${index + 1}: ${file}`);
-    });
-
-    // Create an instance of AbortController
-    const ac = new AbortController();
-    const { signal } = ac; // Get the AbortSignal from the controller
-
-    // Ask the user to choose a file
-    this.replServer.question('Enter the number of the file to open (or type "abort" to cancel): ', { signal }, async (answer) => {
-        // Check if the operation was aborted
-        if (signal.aborted) {
-            console.log('File open operation was aborted.');
+        let files: string[] = undefined;
+        try {
+            // Read all files from the directory
+            files = await fs.promises.readdir(directory);
+        } catch (error) {
+            console.log(`Error reading directory ${directory}: ${error}`);
+            console.log('Changed directory with .cd or .open an/existing/directory');
             this.replServer.displayPrompt();
-            return;
+            return {error: `Error reading directory ${directory}: ${error}`};
         }
+        // Filter out only .json and .yaml files
+        const templateFiles: string[] = files.filter(file => file.endsWith('.json') || file.endsWith('.yaml'));
 
-        const fileIndex = parseInt(answer, 10) - 1; // Convert to zero-based index
-        if (fileIndex >= 0 && fileIndex < templateFiles.length) {
-            // User has entered a valid file number; initialize with this file
-            const filepath = templateFiles[fileIndex];
-            try {
-                const result = await this.init(`-f "${filepath}"`);
-                console.log(StatedREPL.stringify(result));
-                console.log("...try '.out' or 'template.output' to see evaluated template")
-            } catch (error) {
-                console.log('Error loading file:', error);
+        // Display the list of files to the user
+        templateFiles.forEach((file, index) => {
+            console.log(`${index + 1}: ${file}`);
+        });
+
+        // Create an instance of AbortController
+        const ac = new AbortController();
+        const {signal} = ac; // Get the AbortSignal from the controller
+
+        // Ask the user to choose a file
+        this.replServer.question('Enter the number of the file to open (or type "abort" to cancel): ', {signal}, async (answer) => {
+            // Check if the operation was aborted
+            if (signal.aborted) {
+                console.log('File open operation was aborted.');
+                this.replServer.displayPrompt();
+                return;
             }
-        } else {
-            console.log('Invalid file number.');
-        }
 
-        this.replServer.displayPrompt();
-    });
+            const fileIndex = parseInt(answer, 10) - 1; // Convert to zero-based index
+            if (fileIndex >= 0 && fileIndex < templateFiles.length) {
+                // User has entered a valid file number; initialize with this file
+                const filepath = templateFiles[fileIndex];
+                try {
+                    const result = await this.init(`-f "${filepath}"`); // Adjust this call as per your init method's expected format
+                    console.log(StatedREPL.stringify(result));
+                    console.log("...try '.out' or 'template.output' to see evaluated template")
+                } catch (error) {
+                    console.log('Error loading file:', error);
+                }
+            } else {
+                console.log('Invalid file number.');
+            }
+            this.replServer.displayPrompt();
+        });
 
-    // Allow the user to type "abort" to cancel the file open operation
-    this.replServer.once('SIGINT', () => {
-        ac.abort();
-    });
+        // Allow the user to type "abort" to cancel the file open operation
+        this.replServer.once('SIGINT', () => {
+            ac.abort();
+        });
 
-    return "open... (type 'abort' to cancel)";
-}
+        return "open... (type 'abort' to cancel)";
+    }
 
 
     public cd(newDirectory: string) {

diff --git a/src/MetaInfoProducer.ts b/src/MetaInfoProducer.ts
@@ -28,7 +28,7 @@ export default class MetaInfoProducer {
         '\\s*' +                    // Match optional whitespace
         '(?:(@(?<tag>\\w+))?\\s*)' +   // Match the 'tag' like @DEV or @TPC on an expression
         '(?:(?<tempVariable>!)?\\s*)' +    // Match the ! symbol which means 'temp variable'
-        '(?:(?<slash>\\/)|(?<relativePath>(\\.\\.\\/)+))?' + // Match a forward slash '/' or '../' to represent relative paths
+        '(?:(?<slashslash>\\/\\/)|(?<slash>\\/)|(?<relativePath>(\\.\\.\\/)+))?' + // Match a forward slash '/' or '../' to represent relative paths
         '\\$\\{' +                 // Match the literal characters '${'
         '(?<jsonataExpression>[\\s\\S]+)' + // Match one or more of any character. This is the JSONata expression/program (including newline, to accommodate multiline JSONata).
         '\\}' +                    // Match the literal character '}'
@@ -74,9 +74,10 @@ export default class MetaInfoProducer {
                 const keyEndsWithDollars = typeof path[path.length - 1] === 'string' && String(path[path.length - 1]).endsWith('$');
                 const tag = getMatchGroup('tag');
                 const exclamationPoint = !!getMatchGroup('tempVariable');
+                const leadingSlashSlash = getMatchGroup('slashslash');
                 const leadingSlash = getMatchGroup('slash');
                 const leadingCdUp = getMatchGroup('relativePath');
-                const slashOrCdUp = leadingSlash || leadingCdUp;
+                const slashOrCdUp = leadingSlashSlash || leadingSlash || leadingCdUp;
                 const expr = keyEndsWithDollars ? o : getMatchGroup('jsonataExpression');
                 const hasExpression = !!match || keyEndsWithDollars;
 
@@ -104,7 +105,7 @@ export default class MetaInfoProducer {
 
         await getPaths(template);
         return emit;
-        /*
+        /* this is an optimization that may eventually be important to get to
         // Prune subtrees with treeHasExpressions__ = false
         const prunedMetaInfos = fullResult.metaInfos.filter(info => info.treeHasExpressions__);
 

diff --git a/src/StatedREPL.ts b/src/StatedREPL.ts
@@ -121,7 +121,9 @@ export default class StatedREPL {
                 console.log(stringify);
             }
         } catch (e) {
-            console.error(e);
+            const stringify = StatedREPL.stringify(e.message);
+            console.error(stringify);
+            result = "";
         }
         this.r.displayPrompt();
     }