v11.1.7

Please note changes in 11.1.7 may not yet be released in future versions, as
this is a backport and patch release on the 11.1.x series of releases. Updates that
are included in the future will have a corresponding CHANGELOG entry in future
releases.

Fixed

• CUMULUS-3024
  • Update PUT /granules endpoint to operate consistently across datastores
    (PostgreSQL, ElasticSearch, DynamoDB). Previously it was possible, given a
    partial Granule payload to have different data in Dynamo/ElasticSearch and PostgreSQL
  • Given a partial Granule object, the /granules update endpoint now operates
    with behavior more consistent with a PATCH operation where fields not provided
    in the payload will not be updated in the datastores.
  • Granule translation (db/src/granules.ts) now supports removing null/undefined fields when converting from API to Postgres
    granule formats.
  • Update granule write logic: if a null files key is provided in an update payload (e.g. files: null),
    an error will be thrown. null files were not previously supported and would throw potentially unclear errors. This makes the error clearer and more explicit.
  • Update granule write logic: If an empty array is provided for the files key, all files will be removed in all datastores
• CUMULUS-2971
  • Updated @cumulus/aws-client/S3ObjectStore class to take string query parameters and
    its methods signGetObject and signHeadObject to take parameter presignOptions
• CUMULUS-2557
  • Updated @cumulus/aws-client/S3/moveObject to handle zero byte files (0 byte files).
• CUMULUS-3021
  • Updated @cumulus/api-client/collections and @cumulus/integration-tests/api to encode
    collection version in the URI path
• CUMULUS-3027
  • Pinned typescript to ~4.7.x to address typing incompatibility issues
    discussed in knex/knex#5279
  • Update generate-ts-build-cache script to always install root project dependencies

v13.3.0

Notable Changes

• CUMULUS-2930
  • The GET /granules endpoint has a new optional query parameter:
    searchContext, which is used to resume listing within the same search
    context. It is provided in every response from the endpoint as
    meta.searchContext. The searchContext value must be submitted with every
    consequent API call, and must be fetched from each new response to maintain
    the context.
  • Use of the searchContext query string parameter allows listing past 10,000 results.
  • Note that using the from query param in a request will cause the searchContext to
    be ignored and also make the query subject to the 10,000 results cap again.
  • Updated GET /granules endpoint to leverage ElasticSearch search-after API.
    The endpoint will only use search-after when the searchContext parameter
    is provided in a request.

v13.2.1

Notable changes

• CUMULUS-3019
  • Fix file write logic to delete files by granule_cumulus_id instead of
    cumulus_id. Previous logic removed files by matching file.cumulus_id
    to granule.cumulus_id.

v12.0.2

Notable changes

• CUMULUS-3019
  • Fix file write logic to delete files by granule_cumulus_id instead of
    cumulus_id. Previous logic removed files by matching file.cumulus_id
    to granule.cumulus_id.

v11.1.5

Notable changes

• CUMULUS-3019
  • Fix file write logic to delete files by granule_cumulus_id instead of
    cumulus_id. Previous logic removed files by matching file.cumulus_id
    to granule.cumulus_id.

v13.2.0

Changed

• CUMULUS-2940
  • Updated bulk operation lambda to utilize system wide rds_connection_timing
    configuration parameters from the main cumulus module
• CUMULUS-2954
  • Updated Backup LZARDS task to run as a single task in a step function workflow.
  • Updated task to allow user to provide collectionId in workflow input and
    updated task to use said collectionId to look up the corresponding collection record in RDS.

v13.1.0

[v13.1.0] 2022-7-22

MIGRATION notes

• The changes introduced in CUMULUS-2962 will re-introduce a
  files_granules_cumulus_id_index on the files table in the RDS database.
  This index will be automatically created as part of the bootstrap lambda
  function on deployment of the data-persistence module.

  In cases where the index is already applied, this update will have no effect.

  Please Note: In some cases where ingest is occurring at high volume levels and/or the
  files table has > 150M file records, the migration may
  fail on deployment due to timing required to both acquire the table state needed for the
  migration and time to create the index given the resources available.

  For reference a rx.5 large Aurora/RDS database
  with no activity took roughly 6 minutes to create the index for a file table with 300M records and no active ingest, however timed out when the same migration was attempted
  in production with possible activity on the table.

  If you believe you are subject to the above consideration, you may opt to
  manually create the files table index prior to deploying this version of
  Core with the following procedure:

  ---

  • Verify you do not have the index:

  select * from pg_indexes where tablename = 'files';
  
   schemaname | tablename |        indexname        | tablespace |                                       indexdef
  ------------+-----------+-------------------------+------------+---------------------------------------------------------------------------------------
   public     | files     | files_pkey              |            | CREATE UNIQUE INDEX files_pkey ON public.files USING btree (cumulus_id)
   public     | files     | files_bucket_key_unique |            | CREATE UNIQUE INDEX files_bucket_key_unique ON public.files USING btree (bucket, key)
  

  In this instance you should not see an indexname row with
  files_granules_cumulus_id_index as the value. If you do, you should be
  clear to proceed with the installation.

  • Quiesce ingest

  Stop all ingest operations in Cumulus Core according to your operational
  procedures. You should validate that it appears there are no active queries that
  appear to be inserting granules/files into the database as a secondary method
  of evaluating the database system state:

  select pid, query, state, wait_event_type, wait_event from pg_stat_activity where state = 'active';
  

  If query rows are returned with a query value that involves the files table,
  make sure ingest is halted and no other granule-update activity is running on
  the system.

  Note: In rare instances if there are hung queries that are unable to resolve, it may be necessary to
  manually use psql Server Signaling
  Functions
  pg_cancel_backend and/or
  pg_terminate_backend if the migration will not complete in the next step.

  • Create the Index

  Run the following query to create the index. Depending on the situation
  this may take many minutes to complete, and you will note your CPU load and
  disk I/O rates increase on your cluster:

  CREATE INDEX files_granule_cumulus_id_index ON files (granule_cumulus_id);
  

  You should see a response like:

  CREATE INDEX
  

  and can verify the index files_granule_cumulus_id_index was created:

  => select * from pg_indexes where tablename = 'files';
  schemaname | tablename |           indexname            | tablespace |                                           indexdef
   ------------+-----------+--------------------------------+------------+----------------------------------------------------------------------------------------------
   public     | files     | files_pkey                     |            | CREATE UNIQUE INDEX files_pkey ON public.files USING btree (cumulus_id)
   public     | files     | files_bucket_key_unique        |            | CREATE UNIQUE INDEX files_bucket_key_unique ON public.files USING btree (bucket, key)
   public     | files     | files_granule_cumulus_id_index |            | CREATE INDEX files_granule_cumulus_id_index ON public.files USING btree (granule_cumulus_id)
  (3 rows)
  

  • Once this is complete, you may deploy this version of Cumulus as you
    normally would.
    If you are unable to stop ingest for the above procedure and cannot
    migrate with deployment, you may be able to manually create the index while
    writes are ongoing using postgres's CONCURRENTLY option for CREATE INDEX.
    This can have significant impacts on CPU/write IO, particularly if you are
    already using a significant amount of your cluster resources, and may result
    in failed writes or an unexpected index/database state.

  PostgreSQL's
  documentation
  provides more information on this option. Please be aware it is
  unsupported by Cumulus at this time, so community members that opt to go
  this route should proceed with caution.

  ---

Notable changes

• CUMULUS-2962
  • Re-added database structural migration to files table to add an index on granule_cumulus_id
• CUMULUS-2929
  • Updated move-granule task to check the optional collection configuration parameter
    meta.granuleMetadataFileExtension to determine the granule metadata file.
    If none is specified, the granule CMR metadata or ISO metadata file is used.

Changed

• Updated Moment.js package to 2.29.4 to address security vulnerability
• CUMULUS-2967
  • Added fix example/spec/helpers/Provider that doesn't fail deletion 404 in
    case of deletion race conditions

Fixed

• CUMULUS-2995

  • Updated Lerna package to 5.1.8 to address security vulnerability

• CUMULUS-2863

  • Fixed @cumulus/api validateAndUpdateSqsRule method to allow 0 retries and 0 visibilityTimeout
    in rule's meta.

• CUMULUS-2959

  • Fixed @cumulus/api granules module to convert numeric productVolume to string
    when an old granule record is retrieved from DynamoDB

v11.1.4

[v11.1.4] 2022-07-18

MIGRATION notes

• The changes introduced in CUMULUS-2962 will re-introduce a
  files_granules_cumulus_id_index on the files table in the RDS database.
  This index will be automatically created as part of the bootstrap lambda
  function on deployment of the data-persistence module.

  In cases where the index is already applied, this update will have no effect.

  Please Note: In some cases where ingest is occurring at high volume levels and/or the
  files table has > 150M file records, the migration may
  fail on deployment due to timing required to both acquire the table state needed for the
  migration and time to create the index given the resources available.

  For reference a rx.5 large Aurora/RDS database
  with no activity took roughly 6 minutes to create the index for a file table with 300M records and no active ingest, however timed out when the same migration was attempted
  in production with possible activity on the table.

  If you believe you are subject to the above consideration, you may opt to
  manually create the files table index prior to deploying this version of
  Core with the following procedure:

  ---

  • Verify you do not have the index:

  select * from pg_indexes where tablename = 'files';
  
   schemaname | tablename |        indexname        | tablespace |                                       indexdef
  ------------+-----------+-------------------------+------------+---------------------------------------------------------------------------------------
   public     | files     | files_pkey              |            | CREATE UNIQUE INDEX files_pkey ON public.files USING btree (cumulus_id)
   public     | files     | files_bucket_key_unique |            | CREATE UNIQUE INDEX files_bucket_key_unique ON public.files USING btree (bucket, key)
  

  In this instance you should not see an indexname row with
  files_granules_cumulus_id_index as the value. If you do, you should be
  clear to proceed with the installation.

  • Quiesce ingest

  Stop all ingest operations in Cumulus Core according to your operational
  procedures. You should validate that it appears there are no active queries that
  appear to be inserting granules/files into the database as a secondary method
  of evaluating the database system state:

  select pid, query, state, wait_event_type, wait_event from pg_stat_activity where state = 'active';
  

  If query rows are returned with a query value that involves the files table,
  make sure ingest is halted and no other granule-update activity is running on
  the system.

  Note: In rare instances if there are hung queries that are unable to resolve, it may be necessary to
  manually use psql Server Signaling
  Functions
  pg_cancel_backend and/or
  pg_terminate_backend if the migration will not complete in the next step.

  • Create the Index

  Run the following query to create the index. Depending on the situation
  this may take many minutes to complete, and you will note your CPU load and
  disk I/O rates increase on your cluster:

  CREATE INDEX files_granule_cumulus_id_index ON files (granule_cumulus_id);
  

  You should see a response like:

  CREATE INDEX
  

  and can verify the index files_granule_cumulus_id_index was created:

  => select * from pg_indexes where tablename = 'files';
  schemaname | tablename |           indexname            | tablespace |                                           indexdef
   ------------+-----------+--------------------------------+------------+----------------------------------------------------------------------------------------------
   public     | files     | files_pkey                     |            | CREATE UNIQUE INDEX files_pkey ON public.files USING btree (cumulus_id)
   public     | files     | files_bucket_key_unique        |            | CREATE UNIQUE INDEX files_bucket_key_unique ON public.files USING btree (bucket, key)
   public     | files     | files_granule_cumulus_id_index |            | CREATE INDEX files_granule_cumulus_id_index ON public.files USING btree (granule_cumulus_id)
  (3 rows)
  

  • Once this is complete, you may deploy this version of Cumulus as you
    normally would.
    If you are unable to stop ingest for the above procedure and cannot
    migrate with deployment, you may be able to manually create the index while
    writes are ongoing using postgres's CONCURRENTLY option for CREATE INDEX.
    This can have significant impacts on CPU/write IO, particularly if you are
    already using a significant amount of your cluster resources, and may result
    in failed writes or an unexpected index/database state.

  PostgreSQL's
  documentation
  provides more information on this option. Please be aware it is
  unsupported by Cumulus at this time, so community members that opt to go
  this route should proceed with caution.

  ---

Changed

• Updated Moment.js package to 2.29.4 to address security vulnerability

v12.0.1

• CUMULUS-2995
  • Updated Moment.js package to 2.29.4 to address security vulnerability. See GHSA-wc69-rhjr-hc9g

v13.0.1

Fixed

• CUMULUS-2995
  • Updated Moment.js package to 2.29.4 to address security vulnerability. See GHSA-wc69-rhjr-hc9g