Changing content checksums list results in errors and crashloopbackoff

[vkukk]

Version
Please provide the versions of the pulp-operator and pulp images in use.

$ helm -n pulp list
NAME	NAMESPACE	REVISION	UPDATED                                 	STATUS  	CHART              	APP VERSION 
pulp	pulp     	1       	2024-09-03 16:54:04.919341315 +0300 EEST	deployed	pulp-operator-0.1.0	1.0.1-beta.4

apiVersion: repo-manager.pulpproject.org/v1beta2

kind: Pulp

metadata:
  name: pulp
  namespace: pulp

spec:
  allowed_content_checksums:
    - sha1
    - sha256
    - sha512

  object_storage_s3_secret: s3-secret

  database:
    # when changing pulp-postgresql env values, update secret suffix here
    external_db_secret: external-pulp-postgresql-secret-74844b4846

  cache:
    enabled: true
    external_cache_secret: external-pulp-redis-secret-d8d2dhd8bk

  api:
    replicas: 1
    resource_requirements:
      requests:
        cpu: 250m
        memory: 256Mi
      limits:
        cpu: 1
        memory: 512Mi

  content:
    replicas: 1
    resource_requirements:
      requests:
        cpu: 250m
        memory: 256Mi
      limits:
        cpu: 500m
        memory: 512Mi

  worker:
    replicas: 1
    resource_requirements:
      requests:
        cpu: 500m
        memory: 500Mi
      limits:
        cpu: 2
        memory: 1Gi

Describe the bug
Added section to config:

allowed_content_checksums:
    - sha1
    - sha256
    - sha512

Applied changes to Pulp deployment and all api, content and workers are crashing.

$ kubectl -n pulp logs pod/pulp-api-5bb4d46d59-7fcgx 
Waiting on postgresql to start...
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/asgiref/local.py", line 89, in _lock_storage
    asyncio.get_running_loop()
RuntimeError: no running event loop

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 58, in __getitem__
    return getattr(self._connections, alias)
  File "/usr/local/lib/python3.9/site-packages/asgiref/local.py", line 118, in __getattr__
    return getattr(storage, key)
AttributeError: '_thread._local' object has no attribute 'default'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/wait_on_postgres.py", line 12, in <module>
    connection.ensure_connection()
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 15, in __getattr__
    return getattr(self._connections[self._alias], item)
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 60, in __getitem__
    if alias not in self.settings:
  File "/usr/local/lib/python3.9/site-packages/django/utils/functional.py", line 57, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 45, in settings
    self._settings = self.configure_settings(self._settings)
  File "/usr/local/lib/python3.9/site-packages/django/db/utils.py", line 148, in configure_settings
    databases = super().configure_settings(databases)
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 50, in configure_settings
    settings = getattr(django_settings, self.settings_name)
  File "/usr/local/lib/python3.9/site-packages/django/conf/__init__.py", line 102, in __getattr__
    self._setup(name)
  File "/usr/local/lib/python3.9/site-packages/django/conf/__init__.py", line 89, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/local/lib/python3.9/site-packages/django/conf/__init__.py", line 217, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib64/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/settings.py", line 544, in <module>
    raise e
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/settings.py", line 503, in <module>
    raise ImproperlyConfigured(
django.core.exceptions.ImproperlyConfigured: There have been identified artifacts missing checksum 'sha1'. Run 'pulpcore-manager handle-artifact-checksums' first to populate missing artifact checksums.

Expected behavior
Pulp would work and accept listed content checksums.

Additional context
Job to update content checksums is failing

$ kubectl -n pulp logs pod/pulp-update-content-checksums-sxkkw-h86qt 
Waiting on postgresql to start...
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/asgiref/local.py", line 89, in _lock_storage
    asyncio.get_running_loop()
RuntimeError: no running event loop

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 58, in __getitem__
    return getattr(self._connections, alias)
  File "/usr/local/lib/python3.9/site-packages/asgiref/local.py", line 118, in __getattr__
    return getattr(storage, key)
AttributeError: '_thread._local' object has no attribute 'default'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/wait_on_postgres.py", line 12, in <module>
    connection.ensure_connection()
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 15, in __getattr__
    return getattr(self._connections[self._alias], item)
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 60, in __getitem__
    if alias not in self.settings:
  File "/usr/local/lib/python3.9/site-packages/django/utils/functional.py", line 57, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 45, in settings
    self._settings = self.configure_settings(self._settings)
  File "/usr/local/lib/python3.9/site-packages/django/db/utils.py", line 148, in configure_settings
    databases = super().configure_settings(databases)
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 50, in configure_settings
    settings = getattr(django_settings, self.settings_name)
  File "/usr/local/lib/python3.9/site-packages/django/conf/__init__.py", line 102, in __getattr__
    self._setup(name)
  File "/usr/local/lib/python3.9/site-packages/django/conf/__init__.py", line 89, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/local/lib/python3.9/site-packages/django/conf/__init__.py", line 217, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib64/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/settings.py", line 544, in <module>
    raise e
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/settings.py", line 503, in <module>
    raise ImproperlyConfigured(
django.core.exceptions.ImproperlyConfigured: There have been identified artifacts missing checksum 'sha1'. Run 'pulpcore-manager handle-artifact-checksums' first to populate missing artifact checksums.
Checking for database migrations
Traceback (most recent call last):
  File "/usr/local/bin/pulpcore-manager", line 8, in <module>
    sys.exit(manage())
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/manage.py", line 11, in manage
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 416, in execute
    django.setup()
  File "/usr/local/lib/python3.9/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/usr/local/lib/python3.9/site-packages/django/apps/registry.py", line 116, in populate
    app_config.import_models()
  File "/usr/local/lib/python3.9/site-packages/django/apps/config.py", line 269, in import_models
    self.models_module = import_module(models_module_name)
  File "/usr/lib64/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/models/__init__.py", line 34, in <module>
    from .exporter import (
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/models/exporter.py", line 11, in <module>
    from pulpcore.app.models.repository import Repository
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/models/repository.py", line 34, in <module>
    from pulpcore.cache import Cache
  File "/usr/local/lib/python3.9/site-packages/pulpcore/cache/__init__.py", line 1, in <module>
    from .cache import (
  File "/usr/local/lib/python3.9/site-packages/pulpcore/cache/cache.py", line 17, in <module>
    from pulpcore.app.settings import settings
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/settings.py", line 544, in <module>
    raise e
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/settings.py", line 503, in <module>
    raise ImproperlyConfigured(
django.core.exceptions.ImproperlyConfigured: There have been identified artifacts missing checksum 'sha1'. Run 'pulpcore-manager handle-artifact-checksums' first to populate missing artifact checksums.
Database migrated!
Traceback (most recent call last):
  File "/usr/local/bin/pulpcore-manager", line 8, in <module>
    sys.exit(manage())
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/manage.py", line 11, in manage
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/management/commands/handle-artifact-checksums.py", line 179, in handle
    for chunk in fp.chunks(CHUNK_SIZE):
  File "/usr/local/lib/python3.9/site-packages/django/core/files/base.py", line 55, in chunks
    self.seek(0)
  File "/usr/local/lib/python3.9/site-packages/django/core/files/utils.py", line 45, in <lambda>
    seek = property(lambda self: self.file.seek)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/fields/files.py", line 48, in _get_file
    self._file = self.storage.open(self.name, "rb")
  File "/usr/local/lib/python3.9/site-packages/django/core/files/storage/base.py", line 22, in open
    return self._open(name, mode)
  File "/usr/local/lib/python3.9/site-packages/storages/backends/s3.py", line 488, in _open
    f = S3File(name, mode, self)
  File "/usr/local/lib/python3.9/site-packages/storages/backends/s3.py", line 133, in __init__
    self.obj.load(**params)
  File "/usr/local/lib/python3.9/site-packages/boto3/resources/factory.py", line 565, in do_action
    response = action(self, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/boto3/resources/action.py", line 88, in __call__
    response = getattr(parent.meta.client, operation_name)(*args, **params)
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 569, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 1023, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (400) when calling the HeadObject operation: Bad Request

[vkukk]

It all started with attempt to pull content from remote https://repo.mongodb.org/yum/redhat/9/mongodb-org/6.0/x86_64/RPMS/

When attempting to sync repo from that remote:

pulp> rpm repository sync --name mongodb-org-6-tst
Started background task /pulp/api/v3/tasks/0191d739-a109-7ae6-9507-55f2d046797d/
.Error: Task /pulp/api/v3/tasks/0191d739-a109-7ae6-9507-55f2d046797d/ failed: 'Artifact contains forbidden checksum type sha1. You can allow it with 'ALLOWED_CONTENT_CHECKSUMS' setting.'

As adding sha1 fails as described above.

[git-hyagi]

Hi, @vkukk

Thank you for keep providing detailed issue descriptions!
I tried to reproduce this error, but I couldn't.
First, I tried to create a new installation and sync mongo repo through:

pulp rpm repository create --name foo
pulp rpm remote create \
    --name bar \
    --url https://repo.mongodb.org/yum/redhat/9/mongodb-org/6.0/x86_64/
pulp rpm repository update --name foo --remote bar
pulp rpm repository sync --name foo

which, as you mentioned, failed because of the forbidden sha1 checksum.
Then I updated pulp CR with:

  allowed_content_checksums:
    - sha1
    - sha256
    - sha512

which deployed the checksum job and redeployed pulpcore pods, but they all ran fine.

As a second test, I created a new namespace, did another installation of Pulp, and, this time, I tried to first sync a sample RPM repo from pulp:

pulp rpm repository create --name pulp
pulp rpm remote create \
    --name pulp \
    --url 'https://fixtures.pulpproject.org/rpm-unsigned/'
pulp rpm repository update --name pulp --remote pulp
pulp rpm repository sync --name pulp

after that I tried to sync mongo repo (the same way I did before), which failed with the forbidden sha1 checksum again, and when I updated pulp CR I could see the same errors (I'll investigate them later) as you provided in the checksum jobs (for some reason it deployed 2 jobs, I'll also investigate this later), but the redeployed pulpcore (api/content/worker) pods started without errors and re-triggering the sync task worked.

With that being said, do you remember to sync any other repo (before trying to sync mongo repo) or to do something else?

Would you mind checking if this command fixes the error?

kubectl exec deployment/pulp-api -- pulpcore-manager handle-artifact-checksums

[vkukk]

I have managed to reproduce it. When I initially created mongo repo, it had --retain-package-versions 3 --autopublish
I'm using pulp cli shell.

At start:

spec:
 #  allowed_content_checksums:
 #  - sha1
 #  - sha256

rpm remote create --name mongo-6 --url https://repo.mongodb.org/yum/redhat/9/mongodb-org/6.0/x86_64/
rpm repository create --name mongo-6 --remote mongo-6 --retain-package-versions 3 --autopublish
rpm repository sync --name mongo-6
rpm distribution create --name mongo-6-tst --base-path mongo-6/tst --repository mongo-6

Check what is available at https://pulp3.tld/pulp/content/mongo-6/tst/Packages/m/

Decide to remove package version limitation.

rpm repository update --repository mongo-6 --retain-package-versions 0
rpm repository sync --name mongo-6
Started background task /pulp/api/v3/tasks/0191dca2-4a93-7b6c-86fe-381579650f5e/
.Error: Task /pulp/api/v3/tasks/0191dca2-4a93-7b6c-86fe-381579650f5e/ failed: 'Artifact contains forbidden checksum type sha1. You can allow it with 'ALLOWED_CONTENT_CHECKSUMS' setting.'

Now change pulp CR config. Considering that sha256 is default and sync error was about sha1 being forbidden, add sha1 to allowed list.

spec:
  allowed_content_checksums:
    - sha1
    - sha256

Now apply updates to Pulp

kubectl apply -f pulp.yaml

Watch how pulp-update-content checksums jobs fail.

 kubectl -n pulp logs pod/pulp-update-content-checksums-gw2n5-6wcv6
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/asgiref/local.py", line 89, in _lock_storage
    asyncio.get_running_loop()
RuntimeError: no running event loop

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 58, in __getitem__
    return getattr(self._connections, alias)
Waiting on postgresql to start...
  File "/usr/local/lib/python3.9/site-packages/asgiref/local.py", line 118, in __getattr__
    return getattr(storage, key)
AttributeError: '_thread._local' object has no attribute 'default'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/wait_on_postgres.py", line 12, in <module>
    connection.ensure_connection()
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 15, in __getattr__
    return getattr(self._connections[self._alias], item)
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 60, in __getitem__
    if alias not in self.settings:
  File "/usr/local/lib/python3.9/site-packages/django/utils/functional.py", line 57, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 45, in settings
    self._settings = self.configure_settings(self._settings)
  File "/usr/local/lib/python3.9/site-packages/django/db/utils.py", line 148, in configure_settings
    databases = super().configure_settings(databases)
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 50, in configure_settings
    settings = getattr(django_settings, self.settings_name)
  File "/usr/local/lib/python3.9/site-packages/django/conf/__init__.py", line 102, in __getattr__
    self._setup(name)
  File "/usr/local/lib/python3.9/site-packages/django/conf/__init__.py", line 89, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/local/lib/python3.9/site-packages/django/conf/__init__.py", line 217, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib64/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/settings.py", line 544, in <module>
    raise e
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/settings.py", line 503, in <module>
    raise ImproperlyConfigured(
django.core.exceptions.ImproperlyConfigured: There have been identified artifacts missing checksum 'sha1'. Run 'pulpcore-manager handle-artifact-checksums' first to populate missing artifact checksums.
Checking for database migrations
Traceback (most recent call last):
  File "/usr/local/bin/pulpcore-manager", line 8, in <module>
    sys.exit(manage())
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/manage.py", line 11, in manage
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 416, in execute
    django.setup()
  File "/usr/local/lib/python3.9/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/usr/local/lib/python3.9/site-packages/django/apps/registry.py", line 116, in populate
    app_config.import_models()
  File "/usr/local/lib/python3.9/site-packages/django/apps/config.py", line 269, in import_models
    self.models_module = import_module(models_module_name)
  File "/usr/lib64/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/models/__init__.py", line 34, in <module>
    from .exporter import (
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/models/exporter.py", line 11, in <module>
    from pulpcore.app.models.repository import Repository
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/models/repository.py", line 34, in <module>
    from pulpcore.cache import Cache
  File "/usr/local/lib/python3.9/site-packages/pulpcore/cache/__init__.py", line 1, in <module>
    from .cache import (
  File "/usr/local/lib/python3.9/site-packages/pulpcore/cache/cache.py", line 17, in <module>
    from pulpcore.app.settings import settings
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/settings.py", line 544, in <module>
    raise e
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/settings.py", line 503, in <module>
    raise ImproperlyConfigured(
django.core.exceptions.ImproperlyConfigured: There have been identified artifacts missing checksum 'sha1'. Run 'pulpcore-manager handle-artifact-checksums' first to populate missing artifact checksums.
Database migrated!
Traceback (most recent call last):
  File "/usr/local/bin/pulpcore-manager", line 8, in <module>
    sys.exit(manage())
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/manage.py", line 11, in manage
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/management/commands/handle-artifact-checksums.py", line 179, in handle
    for chunk in fp.chunks(CHUNK_SIZE):
  File "/usr/local/lib/python3.9/site-packages/django/core/files/base.py", line 55, in chunks
    self.seek(0)
  File "/usr/local/lib/python3.9/site-packages/django/core/files/utils.py", line 45, in <lambda>
    seek = property(lambda self: self.file.seek)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/fields/files.py", line 48, in _get_file
    self._file = self.storage.open(self.name, "rb")
  File "/usr/local/lib/python3.9/site-packages/django/core/files/storage/base.py", line 22, in open
    return self._open(name, mode)
  File "/usr/local/lib/python3.9/site-packages/storages/backends/s3.py", line 488, in _open
    f = S3File(name, mode, self)
  File "/usr/local/lib/python3.9/site-packages/storages/backends/s3.py", line 133, in __init__
    self.obj.load(**params)
  File "/usr/local/lib/python3.9/site-packages/boto3/resources/factory.py", line 565, in do_action
    response = action(self, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/boto3/resources/action.py", line 88, in __call__
    response = getattr(parent.meta.client, operation_name)(*args, **params)
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 569, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 1023, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (400) when calling the HeadObject operation: Bad Request


$ kubectl -n pulp logs pod/pulp-update-content-checksums-gw2n5-tqghg
Waiting on postgresql to start...
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/asgiref/local.py", line 89, in _lock_storage
    asyncio.get_running_loop()
RuntimeError: no running event loop

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 58, in __getitem__
    return getattr(self._connections, alias)
  File "/usr/local/lib/python3.9/site-packages/asgiref/local.py", line 118, in __getattr__
    return getattr(storage, key)
AttributeError: '_thread._local' object has no attribute 'default'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/wait_on_postgres.py", line 12, in <module>
    connection.ensure_connection()
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 15, in __getattr__
    return getattr(self._connections[self._alias], item)
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 60, in __getitem__
    if alias not in self.settings:
  File "/usr/local/lib/python3.9/site-packages/django/utils/functional.py", line 57, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 45, in settings
    self._settings = self.configure_settings(self._settings)
  File "/usr/local/lib/python3.9/site-packages/django/db/utils.py", line 148, in configure_settings
    databases = super().configure_settings(databases)
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 50, in configure_settings
    settings = getattr(django_settings, self.settings_name)
  File "/usr/local/lib/python3.9/site-packages/django/conf/__init__.py", line 102, in __getattr__
    self._setup(name)
  File "/usr/local/lib/python3.9/site-packages/django/conf/__init__.py", line 89, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/local/lib/python3.9/site-packages/django/conf/__init__.py", line 217, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib64/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/settings.py", line 544, in <module>
    raise e
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/settings.py", line 503, in <module>
    raise ImproperlyConfigured(
django.core.exceptions.ImproperlyConfigured: There have been identified artifacts missing checksum 'sha1'. Run 'pulpcore-manager handle-artifact-checksums' first to populate missing artifact checksums.
Checking for database migrations
Traceback (most recent call last):
  File "/usr/local/bin/pulpcore-manager", line 8, in <module>
    sys.exit(manage())
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/manage.py", line 11, in manage
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 416, in execute
    django.setup()
  File "/usr/local/lib/python3.9/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/usr/local/lib/python3.9/site-packages/django/apps/registry.py", line 116, in populate
    app_config.import_models()
  File "/usr/local/lib/python3.9/site-packages/django/apps/config.py", line 269, in import_models
    self.models_module = import_module(models_module_name)
  File "/usr/lib64/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/models/__init__.py", line 34, in <module>
    from .exporter import (
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/models/exporter.py", line 11, in <module>
    from pulpcore.app.models.repository import Repository
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/models/repository.py", line 34, in <module>
    from pulpcore.cache import Cache
  File "/usr/local/lib/python3.9/site-packages/pulpcore/cache/__init__.py", line 1, in <module>
    from .cache import (
  File "/usr/local/lib/python3.9/site-packages/pulpcore/cache/cache.py", line 17, in <module>
    from pulpcore.app.settings import settings
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/settings.py", line 544, in <module>
    raise e
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/settings.py", line 503, in <module>
    raise ImproperlyConfigured(
django.core.exceptions.ImproperlyConfigured: There have been identified artifacts missing checksum 'sha1'. Run 'pulpcore-manager handle-artifact-checksums' first to populate missing artifact checksums.
Database migrated!
Traceback (most recent call last):
  File "/usr/local/bin/pulpcore-manager", line 8, in <module>
    sys.exit(manage())
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/manage.py", line 11, in manage
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/management/commands/handle-artifact-checksums.py", line 179, in handle
    for chunk in fp.chunks(CHUNK_SIZE):
  File "/usr/local/lib/python3.9/site-packages/django/core/files/base.py", line 55, in chunks
    self.seek(0)
  File "/usr/local/lib/python3.9/site-packages/django/core/files/utils.py", line 45, in <lambda>
    seek = property(lambda self: self.file.seek)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/fields/files.py", line 48, in _get_file
    self._file = self.storage.open(self.name, "rb")
  File "/usr/local/lib/python3.9/site-packages/django/core/files/storage/base.py", line 22, in open
    return self._open(name, mode)
  File "/usr/local/lib/python3.9/site-packages/storages/backends/s3.py", line 488, in _open
    f = S3File(name, mode, self)
  File "/usr/local/lib/python3.9/site-packages/storages/backends/s3.py", line 133, in __init__
    self.obj.load(**params)
  File "/usr/local/lib/python3.9/site-packages/boto3/resources/factory.py", line 565, in do_action
    response = action(self, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/boto3/resources/action.py", line 88, in __call__
    response = getattr(parent.meta.client, operation_name)(*args, **params)
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 569, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 1023, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (400) when calling the HeadObject operation: Bad Request

pulp-api pods crash with:

$ kubectl -n pulp logs pod/pulp-api-5859d66c7-54v58
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/asgiref/local.py", line 89, in _lock_storage
    asyncio.get_running_loop()
RuntimeError: no running event loop

During handling of the above exception, another exception occurred:
Waiting on postgresql to start...

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 58, in __getitem__
    return getattr(self._connections, alias)
  File "/usr/local/lib/python3.9/site-packages/asgiref/local.py", line 118, in __getattr__
    return getattr(storage, key)
AttributeError: '_thread._local' object has no attribute 'default'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/wait_on_postgres.py", line 12, in <module>
    connection.ensure_connection()
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 15, in __getattr__
    return getattr(self._connections[self._alias], item)
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 60, in __getitem__
    if alias not in self.settings:
  File "/usr/local/lib/python3.9/site-packages/django/utils/functional.py", line 57, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 45, in settings
    self._settings = self.configure_settings(self._settings)
  File "/usr/local/lib/python3.9/site-packages/django/db/utils.py", line 148, in configure_settings
    databases = super().configure_settings(databases)
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 50, in configure_settings
    settings = getattr(django_settings, self.settings_name)
  File "/usr/local/lib/python3.9/site-packages/django/conf/__init__.py", line 102, in __getattr__
    self._setup(name)
  File "/usr/local/lib/python3.9/site-packages/django/conf/__init__.py", line 89, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/local/lib/python3.9/site-packages/django/conf/__init__.py", line 217, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib64/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/settings.py", line 544, in <module>
    raise e
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/settings.py", line 503, in <module>
    raise ImproperlyConfigured(
django.core.exceptions.ImproperlyConfigured: There have been identified artifacts missing checksum 'sha1'. Run 'pulpcore-manager handle-artifact-checksums' first to populate missing artifact checksums.

Running suggested command manually:

$ kubectl -n pulp exec deployment/pulp-api -- pulpcore-manager handle-artifact-checksums
Finished aligning checksums with settings.ALLOWED_CONTENT_CHECKSUMS

Yet when I delete now some pod in crashloop, it will continue crashing:

$ kubectl -n pulp logs pod/pulp-api-5859d66c7-nvs6q
Waiting on postgresql to start...
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/asgiref/local.py", line 89, in _lock_storage
    asyncio.get_running_loop()
RuntimeError: no running event loop

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 58, in __getitem__
    return getattr(self._connections, alias)
  File "/usr/local/lib/python3.9/site-packages/asgiref/local.py", line 118, in __getattr__
    return getattr(storage, key)
AttributeError: '_thread._local' object has no attribute 'default'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/wait_on_postgres.py", line 12, in <module>
    connection.ensure_connection()
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 15, in __getattr__
    return getattr(self._connections[self._alias], item)
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 60, in __getitem__
    if alias not in self.settings:
  File "/usr/local/lib/python3.9/site-packages/django/utils/functional.py", line 57, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 45, in settings
    self._settings = self.configure_settings(self._settings)
  File "/usr/local/lib/python3.9/site-packages/django/db/utils.py", line 148, in configure_settings
    databases = super().configure_settings(databases)
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 50, in configure_settings
    settings = getattr(django_settings, self.settings_name)
  File "/usr/local/lib/python3.9/site-packages/django/conf/__init__.py", line 102, in __getattr__
    self._setup(name)
  File "/usr/local/lib/python3.9/site-packages/django/conf/__init__.py", line 89, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/local/lib/python3.9/site-packages/django/conf/__init__.py", line 217, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib64/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/settings.py", line 544, in <module>
    raise e
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/settings.py", line 503, in <module>
    raise ImproperlyConfigured(
django.core.exceptions.ImproperlyConfigured: There have been identified artifacts missing checksum 'sha1'. Run 'pulpcore-manager handle-artifact-checksums' first to populate missing artifact checksums.

[gerrod3]

Can you try the command again with reporting enabled: kubectl -n pulp exec deployment/pulp-api -- pulpcore-manager handle-artifact-checksums --report?