Earthdata Search is a web application developed by NASA EOSDIS to enable data discovery, search, comparison, visualization, and access across EOSDIS' Earth Science data holdings. It builds upon several public-facing services provided by EOSDIS, including the Common Metadata Repository (CMR) for data discovery and access, EOSDIS User Registration System (URS) authentication, the Global Imagery Browse Services (GIBS) for visualization, and a number of OPeNDAP services hosted by data providers.
Copyright © 2007-2024 United States Government as represented by the Administrator of the National Aeronautics and Space Administration. All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
The Earthdata Search application uses Node v14 and Webpack 5 to generate static assets. The serverless application utilizes the following AWS services (important to note if deploying to an AWS environment):
- S3
- We highly recommend using CloudFront in front of S3.
- SQS
- API Gateway
- Lambda
- Cloudwatch (Events)
Earthdata Search runs on Node.js, in order to run the application you'll need to install it.
Recommended: Use Homebrew
brew install node
npm is a separate project from Node.js, and tends to update more frequently. As a result, even if you’ve just downloaded Node.js (and therefore npm), you’ll probably need to update your npm. Luckily, npm knows how to update itself! To update your npm, type this into your terminal:
npm install -g npm@latest
To ensure that you're using the correct version of Node it is recommended that you use Node Version Manager. Installation instructions can be found on the repository. The version used is defined in .nvmrc and will be used automatically if NVM is configured correctly. Using nvm we can switch node versions to the one utilized by Earthdata Search. From the top-level directory:
nvm use
Earthdata Search utilizes the Serverless Framework for managing AWS resources. In order to fully run and manage the application you'll need to install it:
npm install -g serverless@latest
Earthdata Search uses PostgreSQL in production on AWS RDS. If you don't already have it installed, download and install it to your development environment.
Recommended: Use Homebrew
brew install postgresql
Start the PostgreSQL server:
# If you have never used brew services before:
brew tap homebrew/services
# Start the server:
brew services start postgresql
If you decide to install via Homebrew you'll need to create the default user.
createuser -s postgres
Once npm is installed locally, you need to download the dependencies by executing the command below in the project root directory:
npm install
For local development Earthdata Search uses a json configuration file to store secure files, an example is provided and should be copied and completed before attempting to go any further.
cp secret.config.json.example secret.config.json
In order to operate against a local database this file will need dbUsername and dbPassword values set (you may need to update dbHost, dbName or databasePort in static.config.json if you have custom configuration locally).
If you created the postgres user after a new PostgreSQL install as described above, both dbUsername and dbPassword will be the username you use to log into your computer.
Non-secure values are stored in static.config.json. In order to prevent conflicts amongst developers you copy the static config into overrideStatic.config.json and change the config values there. Do not commit changes to static.config.json.
cp static.config.json overrideStatic.config.json
We can configure some of the layouts for the EDSC presentation by updating the defaultPortal value in overrideStatic.config.json. For development purposes we should set this to edsc.
Ensure that you have a database created:
createdb edsc_dev
To run the migrations locally:
DATABASE_URL=postgresql://USERNAME:PASSWORD@localhost:5432/edsc_dev npm run migrate up
Optionally, we can run the migration locally and not within a deployed Lambda. When deployed our database migrations run within Lambda due to the fact that in non-development environments our resources are not publicly accessible. To run the migrations you'll need to invoke the Lambda:
serverless invoke local --function migrateDatabase
The production build of the application will be output in the /static/dist/ directory:
npm run build
This production build can be run locally with any number of http-server solutions. A simple one is to use the http-server package
npx http-server static/dist
The local development environment for the static assets can be started by executing the command below in the project root directory:
npm run start
This will run the React application at http://localhost:8080 -- please see Serverless Framework below for enabling the 'server' side functionality.
The serverless framework offers many plugins which allow for local development utilizing many of the services AWS offers. For the most part we only need API Gateway and Lambda for this application but there are plugins for many more services (a list of known exceptions will be maintained below).
- SQS
While there is an sqs-offline plugin for serverless it still requires an actual queue be running, we may investigate this in the future but for now sqs functionality isn't available while developing locally which means the following pieces of functionality will not operate locally:
-
Generating Colormaps
-
Scale images
Scaling thumbnail images utilizes a redis cache in the deployed environment. To utilize this cache locally you'll need to install Redis on the dev machine. The easiest way to do this would be by running it in a docker container using the command npm run start:cache. You can also use a visualizer such as RedisInsight to more easily inspect the cache. You will also need to set the environment variable USE_CACHE locally to true with export USE_CACHE=true or add the environment variable to your shell script. To stop the docker container use the npm run stop:cache command.
Running the following command will spin up API Gateway and Lambda locally which will open up a vast majority of the functionality the backend offers.
npm run offline
This will provide access to API Gateway at http://localhost:3001
Additionally, this ties in with the serverless webpack plugin which will ensure that your lambdas are re-built when changes are detected.
Run the Automated Jest tests
Once the project is built, you must ensure that the automated unit tests pass:
npm run test
Run the Automated Cypress tests
You must also ensure that the automated integration tests pass:
npm run cypress:run
You can also use the Cypress GUI with:
npm run cypress:open
When adding new Cypress tests, you will need to modify the secrets.config.json file. You will need to edit the "cypress" object to include data from your local database:
"cypress": {
"user": {
"id": 1, // This should match the ID of your user in the 'users' database table
"username": "your username here" // Replace with the urs_id field of your user in the 'users' database table
}
}
When the time comes to deploy the application, first ensure that you have the required ENV vars set:
- AWS_ACCESS_KEY_ID
- AWS_SECRET_ACCESS_KEY
This application runs in a VPC for NASA security purposes, therefore the following values are expected when a deployment occurs:
- VPC_ID
- SUBNET_ID_A
- SUBNET_ID_B
For production use, this application uses Scatter Swap to obfuscate some IDs -- the library does not require a value be provided but if you'd like to control it you can set the following ENV vars:
- OBFUSCATION_SPIN
- OBFUSCATION_SPIN_SHAPEFILES
To deploy the full application use the following:
NODE_ENV=production serverless deploy --stage UNIQUE_STAGE