Migration error upgrading from v2.9.1 to v4.1.2 #592

glennmatthews · 2024-10-15T15:09:35Z

Describe the problem

We're seeing an issue upgrading an existing deployment from Constance 2.9.1 to 4.1.2.

Steps to reproduce

Have a project with database records from 2.9.1 (Constance configs in constance_config database table, migrations run include database 0001_initial and database 0002_auto_20190129_2304
Have apps in the project that attempt to read from the Constance config at startup time (e.g. to determine which app features are administratively enabled)
Upgrade to Constance 4.1.2 and run migrations

04:22:08.517 WARNING nautobot.apps.config :
  "PLUGINS_CONFIG['nautobot_chatops']['enable_grafana']" is not in settings, and could not read from the Constance datab
ase table (perhaps not initialized yet?)
...
04:22:17.282 WARNING nautobot.core.utils.config :
  Configuration "NETWORK_DRIVERS" is not in settings, and could not read from the Constance database table (perhaps not 
initialized yet?)
Performing database migrations...
...
Applying constance.0001_initial... OK
  Applying constance.0002_migrate_from_old_table...Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "constance_constance_pkey"
DETAIL:  Key (id)=(1) already exists.


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/nautobot-server", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.11/site-packages/nautobot/core/cli/__init__.py", line 293, in main
    execute_from_command_line([sys.argv[0], *unparsed_args])
  File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_l
ine
    utility.execute()
  File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/nautobot/core/management/commands/post_upgrade.py", line 91, in handle
    call_command(
  File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 194, in call_command
    return command.execute(*args, **defaults)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 106, in wrapper
    res = handle_func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/management/commands/migrate.py", line 356, in handle
    post_migrate_state = executor.migrate(
                         ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/migrations/executor.py", line 135, in migrate
    state = self._migrate_all_forwards(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards
    state = self.apply_migration(
            ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/migrations/executor.py", line 252, in apply_migration
    state = migration.apply(state, schema_editor)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/migrations/migration.py", line 132, in apply
    operation.database_forwards(
  File "/usr/local/lib/python3.11/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwa
rds
    self.code(from_state.apps, schema_editor)
  File "/usr/local/lib/python3.11/site-packages/constance/migrations/0002_migrate_from_old_table.py", line 21, in _migra
te_from_old_table
    cursor.execute(
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 84, in _execute
    with self.db.wrap_database_errors:
  File "/usr/local/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
django.db.utils.IntegrityError: duplicate key value violates unique constraint "constance_constance_pkey"
DETAIL:  Key (id)=(1) already exists.

Sure enough, somehow the attempt to read the records from constance_constance during app startup has resulted in records being created in this newly initialized table, which are now conflicting with the records that Constance is trying to auto-migrate from constance_config:

nautobot=# select id, key from constance_config;
 id |                             key                              
----+--------------------------------------------------------------
  1 | constance:nautobot:DEPLOYMENT_ID
  2 | constance:nautobot:DYNAMIC_GROUPS_MEMBER_CACHE_TIMEOUT
  3 | constance:nautobot:nautobot_chatops__enable_grafana
  4 | constance:nautobot:nautobot_chatops__enable_slack
  5 | constance:nautobot:nautobot_chatops__enable_ms_teams
  6 | constance:nautobot:nautobot_chatops__enable_webex
  7 | constance:nautobot:nautobot_chatops__enable_mattermost
  8 | constance:nautobot:SUPPORT_MESSAGE
  9 | constance:nautobot:CHANGELOG_RETENTION
 10 | constance:nautobot:RELEASE_CHECK_URL
 11 | constance:nautobot:BANNER_TOP
 12 | constance:nautobot:BANNER_BOTTOM
 13 | constance:nautobot:PAGINATE_COUNT
 14 | constance:nautobot:MAX_PAGE_SIZE
 15 | constance:nautobot:PER_PAGE_DEFAULTS
 16 | constance:nautobot:nautobot_chatops__enable_ansible
 17 | constance:nautobot:nautobot_chatops__enable_aci
 18 | constance:nautobot:nautobot_chatops__enable_ipfabric
 19 | constance:nautobot:nautobot_chatops__enable_aristacv
 20 | constance:nautobot:nautobot_chatops__enable_meraki
 21 | constance:nautobot:nautobot_chatops__enable_panorama
 22 | constance:nautobot:NETWORK_DRIVERS
 23 | constance:nautobot:PREFER_IPV4
 24 | constance:nautobot:DEVICE_NAME_AS_NATURAL_KEY
 25 | constance:nautobot:LOCATION_NAME_AS_NATURAL_KEY
 26 | constance:nautobot:nautobot_golden_config__DEFAULT_FRAMEWORK
 27 | constance:nautobot:BANNER_LOGIN
 28 | constance:nautobot:RACK_ELEVATION_DEFAULT_UNIT_WIDTH
 29 | constance:nautobot:RACK_ELEVATION_DEFAULT_UNIT_HEIGHT
(29 rows)

nautobot=# select id, key from constance_constance;
 id |                          key                           
----+--------------------------------------------------------
  1 | constance:nautobot:nautobot_chatops__enable_grafana
  2 | constance:nautobot:NETWORK_DRIVERS
  3 | constance:nautobot:nautobot_chatops__enable_slack
  4 | constance:nautobot:nautobot_chatops__enable_ms_teams
  5 | constance:nautobot:nautobot_chatops__enable_webex
  6 | constance:nautobot:nautobot_chatops__enable_mattermost
(6 rows)

All of the records currently showing in constance_constance are settings that apps are attempting to read during application startup, so something (presumably django-constance itself?) resulted in these being written to the database in the middle of running migrations.

System configuration

Django version: 4.2
Python version: 3.11
Django-Constance version: 4.1.2

The text was updated successfully, but these errors were encountered:

Mogost · 2024-11-04T13:27:38Z

You're trying to do a massive upgrade. Including v2 -> v3 and v3 -> v4 at the same time.
From the description, it's difficult to determine where is a mistake and how we can help you.
I recommend you to upgrade step-by-step to simplify the process in general

glennmatthews · 2024-11-04T17:35:21Z

To be clear, migrating the same data from v2 to v3 is successful. It's only in trying to migrate from v2 directly to v4 that it fails.

Mogost · 2024-11-04T18:15:54Z

If you have already migrated to v3 which errors do you have v3->v4?

glennmatthews · 2024-11-04T18:47:31Z

Migrating v3 to v4 also seems to be successful. It's only the direct upgrade from v2 to v4 that fails. I'm guessing the issue may be with the ways the migrations for constance have been outright rewritten between major releases, instead of being left as-is and augmented by new migration files?

We have users who are still running an older version of our code that uses constance v2. We'd like to make the latest version of our code use constance v4. If we have to make upgrading users make a "stop" along the way at an intermediate version of our code that uses constance v3, we can write appropriate documentation if necessary, but it will be something some deployments stumble over so I'd prefer to avoid that if possible.

Mogost · 2024-11-05T10:42:35Z

This looks strange to me.
Do you have a way to avoid constance calls before migrations?

glennmatthews · 2024-11-05T13:47:02Z

May be possible, the issue is that we have apps that are relying on the Constance config data at app initialization to determine whether certain functionality should be enabled/disabled. I'm not the primary author of those apps so I'll have to do some investigation to determine how difficult it would be to have them defer their Constance interaction to a later point.

Thanks!

Mogost · 2024-11-05T13:55:28Z

@glennmatthews I believe you can change https://github.com/nautobot/nautobot/blob/eaa693abcf860fd98276c366c36ef01c6493f058/nautobot/core/utils/config.py#L7-L12 this function, all you need is ensure is the Constance already migrated to the right version or not before accessing config..

glennmatthews · 2024-11-05T13:59:06Z

Hmm, possibly - What would be the appropriate way to make that determination?

glennmatthews · 2024-11-05T14:06:52Z

btw here (https://github.com/nautobot/nautobot/blob/next/nautobot/core/utils/config.py) is the updated version of that file that tries to work around constance unavailability during startup.

Mogost · 2024-11-05T14:10:48Z

What would be the appropriate way to make that determination?

There are many ways to do this. I think the easiest is to check if the 0003_drop_pickle is applied. Other solutions will require raw reading from tables and might be more difficlult.

glennmatthews mentioned this issue Oct 15, 2024

Move django-constance back to v3 nautobot/nautobot#6371

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migration error upgrading from v2.9.1 to v4.1.2 #592

Migration error upgrading from v2.9.1 to v4.1.2 #592

glennmatthews commented Oct 15, 2024

Mogost commented Nov 4, 2024

glennmatthews commented Nov 4, 2024

Mogost commented Nov 4, 2024

glennmatthews commented Nov 4, 2024

Mogost commented Nov 5, 2024

glennmatthews commented Nov 5, 2024

Mogost commented Nov 5, 2024

glennmatthews commented Nov 5, 2024

glennmatthews commented Nov 5, 2024

Mogost commented Nov 5, 2024

Migration error upgrading from v2.9.1 to v4.1.2 #592

Migration error upgrading from v2.9.1 to v4.1.2 #592

Comments

glennmatthews commented Oct 15, 2024

Describe the problem

Steps to reproduce

System configuration

Mogost commented Nov 4, 2024

glennmatthews commented Nov 4, 2024

Mogost commented Nov 4, 2024

glennmatthews commented Nov 4, 2024

Mogost commented Nov 5, 2024

glennmatthews commented Nov 5, 2024

Mogost commented Nov 5, 2024

glennmatthews commented Nov 5, 2024

glennmatthews commented Nov 5, 2024

Mogost commented Nov 5, 2024