-
Notifications
You must be signed in to change notification settings - Fork 78
How modules are loaded
By default, ml-gradle loads modules from src/main/ml-modules. This directory aligns with the Maven convention of placing all application code under "src/main", and "ml-modules" was chosen as a MarkLogic-specific path.
To accommodate storing both REST API specific modules (such as options, services, and transforms) and "regular" application modules under the same parent directory, the following directory structure is used for identifying where different kinds of modules should be stored:
| Path | Description |
|---|---|
| /root | Store any modules here that aren't REST API options, services, transforms, or namespaces. The module will be loaded with a URI relative to "/root" (it won't have "/root" in the URI). |
| /ext | You can also store any modules here too, just like "/root". What's the difference? "/ext" was actually supported first in an effort to mirror what the REST API suggests with its /v1/ext endpoint. But in practice, there's little value in storing modules under "/ext", so it's easier just to toss everything under "/root". |
| /options | REST API search options are stored here. The name of the file (minus the extension) is used to name the search options loaded into ML. |
| /services | REST API resource extensions are stored here. The name of the file (minus the extension) is used to name the resource extension. |
| /transforms | REST API transforms are stored here. The name of the file (minus the extension) is used to name the transform. |
| /namespaces | A REST API namespace namespace can be registered via a file containing the namespace URI, and the name of the file (minus the extension, which can be anything) is used for the namespace prefix. This has been deprecated in the server and support for it in ml-gradle is being removed in the 5.0 release. The recommended way is to define namespaces in a server payload file. |
| /rest-properties.json or /rest-properties.xml | If either file exists, it's used to configure the REST API server associated with mlRestPort/mlTestRestPort via the REST properties endpoint. |
Modules are loaded when you run "mlDeploy". They can also be loaded by themselves (note that prior to ml-gradle 4.0.0, mlLoadModules only loads modules that have been created or modified since mlLoadModules was last run - mlReloadModules ensures everything is loaded; starting with 4.0.0, mlLoadModules loads all modules by default):
gradle mlLoadModules
It's often helpful to use info-level logging so you can see which modules are loaded:
gradle -i mlLoadModules
And debug-level logging will show all the directories that ml-gradle looks in for modules:
gradle -d mlLoadModules
You can also reload modules - i.e. clear the modules database first, and then load modules:
gradle mlReloadModules
If you run into any problems, please see Debugging module loading.
ml-gradle has to handle REST modules - options, transforms, services, and namespaces - differently from non-REST modules, as REST modules must be loaded via an app-specific REST server. However, not every MarkLogic app has a REST server. For this reason, ml-gradle takes the following approach by default:
- Non-REST modules are loaded via the port defined by
mlAppServicesPort, which defaults to 8000. This port is nearly guaranteed to exist in every ML cluster, making it a safe choice. - REST modules are loaded via the port defined by
mlRestPort, which does not have a default value.
See the Property reference for all of the properties that are used for connecting to these ports. Also see Configuring security for how to configure a user for the app server identified by mlAppServicesPort and the app server identified by mlRestPort. Note in particular the need for the xdmp-eval-in privilege for the user specified by mlAppServicesUsername. This privilege can be granted to the user via the rest-evaluator role, the security role, or via a custom role.
This distinction between app servers is important for when you run into an error while loading modules, as your non-REST modules may be loading fine, but your REST modules are not.
Finally, if your app servers are requiring SSL, you'll need to configure that for both the App-Services server and your REST server. In a simple scenario, that will involve the following properties being set:
mlAppServicesSimpleSsl=true
mlSimpleSsl=true
See Loading modules via SSL for information on more complex SSL scenarios.
As of 3.0.0, custom collections and permissions can be specified for each directory of modules under src/main/ml-modules/root (this does not apply to options, services, or transforms, as the MarkLogic REST API applies specific collections and permissions to those modules). This is accomplished via files names collections.properties and permissions.properties and is identical to how those files work for loading config, data, and schemas. See the documentation on loading files for more information.
As of 4.6.0, the properties mlCascadeCollections and mlCascadePermissions can be set to true so that the settings in collections.properties and permissions.properties will be applied to child directories, unless a child directory has its own files.
Note that in the absence of a permissions.properties file for a directory of modules, the value of mlModulePermissions will be used to set the permissions for each file in the directory.
As described in the Configuring resources page, a "custom tokens" map can be populated to specify tokens in resource payloads that will be replaced. These tokens will also be replaced in module files - but as of version 3.2.0, this is only done in "asset" modules, not yet REST API modules (services, options, and transforms).
Two properties control this behavior that are worth noting:
- mlReplaceTokensInModules - controls whether tokens are replaced or not, defaults to true
- mlUseRoxyTokenPrefix - (see the note below about how this property changes in 3.2.0) expects tokens to start with "@ml.", mirroring Roxy's behavior. This defaults to true; if you don't need these prefixes in place, be sure to set this property to false.
NEW in version 3.2.0 - as described on the Configuring resources page, all Gradle properties will be added to the tokens map by default. And, mlUseRoxyTokenPrefix now defaults to false, effectively deprecating that feature. You can emulate that behavior by setting the following properties to mirror Roxy's behavior:
mlTokenPrefix=@ml.
mlTokenSuffix=
Thus, in 3.2.0, with mlPropsAsTokens and mlReplaceTokensInModules both defaulting to true, the default behavior is that all Gradle properties are used as tokens (with "%%" as the default prefix and suffix), and these tokens will be replaced in modules if they exist.
You can see exactly which tokens will be replaced, along with what value, by running the following task:
gradle mlPrintTokens
See Watching for module changes for more information on this topic.
On a project with REST extensions, loading all of an application's modules can take many seconds, as each REST extension must be loaded in a separate call to the MarkLogic Client REST API. This isn't too painful for initializing an empty modules database, but it's unacceptable during a code-test cycle.
To address this, ml-gradle keeps track of when it last loaded each module. This is done via a properties file in the "build" directory of the project (so that the properties file is not in version control). The path of the file is build/ml-javaclient-util/module-timestamp.properties. When ml-gradle tries to load a module, it first checks to see if the module's last-modified timestamp is greater than what is recorded in the properties file. If so, the module is loaded, and the properties file is updated. If not, the module is not loaded.
As of 3.3.1, this feature can be disabled by setting the mlModuleTimestampsPath property to an empty string. Prior to 3.3.1, it can be disabled in the following manner in build.gradle:
ext {
mlAppConfig.setModuleTimestampsPath(null)
}
In practice though, it's common that you'll either run "mlReloadModules" to clear a modules database and load all modules (which ensures that any modules deleted from a project are also deleted from the modules database), or you'll run "mlWatch" to load files as they're created/modified.
Starting in 3.14.0, ml-gradle will include the name of the host that modules are loaded via in the property keys in the module timestamps file. This allows you to e.g. run mlLoadModules against different environments from the same build and still get modules loaded into each modules database. If you wish to disable this behavior, just set the following property in gradle.properties:
mlModuleTimestampsUseHost=false
Starting with 4.0.0 though, this feature typically will not be needed since mlLoadModules loads all modules by default, where prior to 4.0.0, only new/modified modules were loaded by default.
Search options are different from other REST extensions in that they are only made available via the REST server that was used to load them. So for example, if you have a DHF project, and you have a set of search options you'd like to be available via both the staging and final REST servers - well, there's not an easy way to do that. The options you define under src/main/ml-modules/options will be loaded via the final REST server, and thus not available via the staging REST server.
Version 3.12.0 of ml-gradle introduces a new task for copying search options to address this problem - CopySearchOptionsTask:
task copyMyOptions(type: com.marklogic.gradle.task.client.CopySearchOptionsTask) {
mustRunAfter mlLoadModules // This ensures that when running mlReloadModules, modules are loaded first
sourceServer = "data-hub-FINAL"
targetServer = "data-hub-STAGING"
optionsFilename = "my-options.json"
// Can set a "group" property if the servers do not belong to the Default group
// Can set a "client" property to use a DatabaseClient other than the one returned by mlAppConfig.newModulesDatabaseClient()
}
After defining an instance of CopySearchOptionsTask (can define many as needed), you can attach it to the end of a deployment:
mlPostDeploy.dependsOn copyMyOptions
Unfortunately there's not an easy way to integrate this into mlWatch, nor attach it to mlLoadModules or mlReloadModules. But at least for the latter, you can always run both task:
gradle mlReloadModules copyMyOptions
Or just make a custom task that calls those together.
The Client REST API allows for metadata to be defined when loading resources and transforms. See Service metadata in the ml-javaclient-util project for more information.