Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CUMULUS-3847: Remove any remaining ES indexing functions and tests #3897

Open
wants to merge 14 commits into
base: feature/es-phase-2
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 7 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,8 @@ External tooling making use of `searchContext` in the `GET` `/granules/` endpoin
- Updated `@cumulus/api/bin/serveUtils` to no longer add records to ElasticSearch
- Removed ElasticSearch from local API server code
- Updated CollectionSearch to filter granule fields in addition to time frame for active collections
- **CUMULUS-3847**
- list changes here
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

placeholder, I'm still making sure I've captured everything. will make sure this is filled out when I'm fairly certain


## [Unreleased]

Expand Down
5 changes: 0 additions & 5 deletions example/deployments/cumulus/sandbox.tfvars
Original file line number Diff line number Diff line change
Expand Up @@ -45,11 +45,6 @@ csdap_host_url = "https://auth.csdap.uat.earthdatacloud.nasa.gov"

default_s3_multipart_chunksize_mb = 128

elasticsearch_client_config = {
create_reconciliation_report_es_scroll_duration = "8m"
create_reconciliation_report_es_scroll_size = 1500
}

launchpad_api = "https://api.launchpad.nasa.gov/icam/api/sm/v1"
launchpad_certificate = "launchpad.pfx"

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ describe('The AsyncOperation task runner executing a failing lambda function', (
id: asyncOperationId,
taskArn: randomString(),
description: 'Some description',
operationType: 'ES Index',
operationType: 'Bulk Granules',
status: 'RUNNING',
};

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ describe('The AsyncOperation task runner running a non-existent lambda function'
id: asyncOperationId,
taskArn: randomString(),
description: 'Some description',
operationType: 'ES Index',
operationType: 'Bulk Granules',
status: 'RUNNING',
};

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ describe('The AsyncOperation task runner with a non-existent payload', () => {
id: asyncOperationId,
taskArn: randomString(),
description: 'Some description',
operationType: 'ES Index',
operationType: 'Bulk Granules',
status: 'RUNNING',
};

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ describe('The AsyncOperation task runner with a non-JSON payload', () => {
id: asyncOperationId,
taskArn: randomString(),
description: 'Some description',
operationType: 'ES Index',
operationType: 'Kinesis Replay',
status: 'RUNNING',
};

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ describe('The AsyncOperation task runner executing a successful lambda function'

const asyncOperationObject = {
description: 'Some description',
operationType: 'ES Index',
operationType: 'Bulk Granules',
id: asyncOperationId,
taskArn: randomString(),
status: 'RUNNING',
Expand Down
3 changes: 0 additions & 3 deletions packages/api/app/routes.js
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,6 @@ const stats = require('../endpoints/stats');
const version = require('../endpoints/version');
const workflows = require('../endpoints/workflows');
const dashboard = require('../endpoints/dashboard');
const elasticsearch = require('../endpoints/elasticsearch');
const deadLetterArchive = require('../endpoints/dead-letter-archive');
const { launchpadProtectedAuth } = require('./launchpadAuth');
const launchpadSaml = require('../endpoints/launchpadSaml');
Expand Down Expand Up @@ -110,8 +109,6 @@ router.delete('/tokenDelete/:token', token.deleteTokenEndpoint);

router.use('/dashboard', dashboard);

router.use('/elasticsearch', ensureAuthorized, elasticsearch.router);

// Catch and send the error message down (instead of just 500: internal server error)
router.use(defaultErrorHandler);

Expand Down
2 changes: 1 addition & 1 deletion packages/api/ecs/async-operation/tests/test-index.js
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ test.beforeEach(async (t) => {
t.context.testAsyncOperation = {
id: t.context.asyncOperationId,
description: 'test description',
operationType: 'ES Index',
operationType: 'Reconciliation Report',
status: 'RUNNING',
createdAt: Date.now(),
};
Expand Down
233 changes: 0 additions & 233 deletions packages/api/endpoints/elasticsearch.js

This file was deleted.

35 changes: 0 additions & 35 deletions packages/api/lambdas/bootstrap.js

This file was deleted.

4 changes: 2 additions & 2 deletions packages/api/lambdas/bulk-operation.js
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ async function applyWorkflowToGranules({
* @param {number} [payload.concurrency]
* granule concurrency for the bulk deletion operation. Defaults to 10
* @param {Object} [payload.query] - Optional parameter of query to send to ES
* @param {string} [payload.index] - Optional parameter of ES index to query.
* @param {string} [payload.index] - Optional parameter of ES index (metrics) to query.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Meaning we're checking the Cloud Metrics ES index?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, my understanding of why we need to keep the ES related functionality is that for some of the granules and bulk operations it will still query Cloud Metrics for data. I don't feel like totally understand how it all ties together, but that is what is mentioned in code and what I was informed of. So I thought it would be helpful to clarify as much as possible that the ES indexing and querying is going to Cloud Metrics and that's why it is staying around and not being removed as part of ES removal.

* Must exist if payload.query exists.
* @param {Object} [payload.granules] - Optional list of granule unique IDs to bulk operate on
* e.g. { granuleId: xxx, collectionID: xxx }
Expand Down Expand Up @@ -179,7 +179,7 @@ async function bulkGranuleDelete(
* @param {Object} [payload.meta] - Optional meta to add to workflow input
* @param {string} [payload.queueUrl] - Optional name of queue that will be used to start workflows
* @param {Object} [payload.query] - Optional parameter of query to send to ES
* @param {string} [payload.index] - Optional parameter of ES index to query.
* @param {string} [payload.index] - Optional parameter of ES index (metrics) to query.
* Must exist if payload.query exists.
* @param {Object} [payload.granules] - Optional list of granule unique IDs to bulk operate on
* e.g. { granuleId: xxx, collectionID: xxx }
Expand Down
Loading