Skip to content
This repository was archived by the owner on Sep 12, 2024. It is now read-only.

[BUG] Prefetch metadata sync status goes to "Error" due to Timeout issue #894

Open
rjdp opened this issue Apr 6, 2022 · 0 comments
Open

[BUG] Prefetch metadata sync status goes to "Error" due to Timeout issue #894

rjdp opened this issue Apr 6, 2022 · 0 comments
Labels
🐛 bug Something isn't working

Comments

@rjdp
Copy link
Contributor

rjdp commented Apr 6, 2022

Describe the bug

Based on Worker logs we found that Sync status for some Postgres datasource went into "Error" state.

Explain the environment

  • Chaos Genius version: Release candidate 0.6.0
  • OS Version / Instance: Ubuntu 18.04, AWS EC2
  • Deployment type: Docker

Current behavior

  • Intermittently reproducible
  • goes into "Error" state when this happens

Expected behavior

  • Should not Error out and continue to progress eventually completing syncup and moving to "Live/completed" state

Screenshots

image

Additional context

Logs

  • SSL SYSCALL error: EOF detected
{"log":"{\"asctime\": \"2022-04-04 06:50:17,140\", \"levelname\": \"ERROR\", \"name\": \"chaos_genius.controllers.data_source_metadata_controller\", \"message\": \"Error in metadata prefetch.\", \"lineno\": 160, \"funcName\": \"run_metadata_prefetch\", \"filename\": \"data_source_metadata_controller.py\", \"exc_info\": \"Traceback (most recent call last):\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\\\", line 1770, in _execute_context\\n    self.dialect.do_execute(\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py\\\", line 717, in do_execute\\n    cursor.execute(statement, parameters)\\npsycopg2.OperationalError: SSL SYSCALL error: EOF detected\\n\\n\\nThe above exception was the direct cause of the following exception:\\n\\nTraceback (most recent call last):\\n  File \\\"/usr/src/app/chaos_genius/controllers/data_source_metadata_controller.py\\\", line 146, in run_metadata_prefetch\\n    _ = scan_db_and_save_table_info(\\n  File \\\"/usr/src/app/chaos_genius/controllers/data_source_metadata_controller.py\\\", line 208, in scan_db_and_save_table_info\\n    table_info = get_table_info(\\n  File \\\"/usr/src/app/chaos_genius/connectors/__init__.py\\\", line 80, in get_table_info\\n    table_info[\\\"columns\\\"] = db_connection.get_columns(table_name, schema)\\n  File \\\"/usr/src/app/chaos_genius/connectors/base_db.py\\\", line 114, in get_columns\\n    db_columns = self.inspector.get_columns(table_name=use_table, schema=use_schema)\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/reflection.py\\\", line 497, in get_columns\\n    col_defs = self.dialect.get_columns(\\n  File \\\"\u003cstring\u003e\\\", line 2, in get_columns\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/reflection.py\\\", line 55, in cache\\n    ret = fn(self, con, *args, **kw)\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/dialects/postgresql/base.py\\\", line 3652, in get_columns\\n    domains = self._load_domains(connection)\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/dialects/postgresql/base.py\\\", line 4420, in _load_domains\\n    c = connection.execution_options(future_result=True).execute(s)\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\\\", line 1262, in execute\\n    return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py\\\", line 329, in _execute_on_connection\\n    return connection._execute_clauseelement(\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\\\", line 1451, in _execute_clauseelement\\n    ret = self._execute_context(\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\\\", line 1813, in _execute_context\\n    self._handle_dbapi_exception(\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\\\", line 1994, in _handle_dbapi_exception\\n    util.raise_(\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py\\\", line 211, in raise_\\n    raise exception\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\\\", line 1770, in _execute_context\\n    self.dialect.do_execute(\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py\\\", line 717, in do_execute\\n    cursor.execute(statement, parameters)\\nsqlalchemy.exc.OperationalError: (psycopg2.OperationalError) SSL SYSCALL error: EOF detected\\n\\n[SQL: \\n            SELECT t.typname as \\\"name\\\",\\n               pg_catalog.format_type(t.typbasetype, t.typtypmod) as \\\"attype\\\",\\n               not t.typnotnull as \\\"nullable\\\",\\n               t.typdefault as \\\"default\\\",\\n               pg_catalog.pg_type_is_visible(t.oid) as \\\"visible\\\",\\n               n.nspname as \\\"schema\\\"\\n            FROM pg_catalog.pg_type t\\n               LEFT JOIN pg_catalog.pg_namespace n ON n.oid = t.typnamespace\\n            WHERE t.typtype = 'd'\\n        ]\\n(Background on this error at: http://sqlalche.me/e/14/e3q8)\"}\n","stream":"stderr","time":"2022-04-04T06:50:17.141336642Z"}
{"log":"[2022-04-04 06:50:17,140: ERROR/ForkPoolWorker-2] Error in metadata prefetch.\n","stream":"stderr","time":"2022-04-04T06:50:17.141533862Z"}
{"log":"Traceback (most recent call last):\n","stream":"stderr","time":"2022-04-04T06:50:17.141543748Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\", line 1770, in _execute_context\n","stream":"stderr","time":"2022-04-04T06:50:17.141547734Z"}
{"log":"    self.dialect.do_execute(\n","stream":"stderr","time":"2022-04-04T06:50:17.141552351Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py\", line 717, in do_execute\n","stream":"stderr","time":"2022-04-04T06:50:17.141556126Z"}
{"log":"    cursor.execute(statement, parameters)\n","stream":"stderr","time":"2022-04-04T06:50:17.141560305Z"}
{"log":"psycopg2.OperationalError: SSL SYSCALL error: EOF detected\n","stream":"stderr","time":"2022-04-04T06:50:17.141563804Z"}
{"log":"\n","stream":"stderr","time":"2022-04-04T06:50:17.141567412Z"}
{"log":"\n","stream":"stderr","time":"2022-04-04T06:50:17.141571224Z"}
{"log":"The above exception was the direct cause of the following exception:\n","stream":"stderr","time":"2022-04-04T06:50:17.141575141Z"}
{"log":"\n","stream":"stderr","time":"2022-04-04T06:50:17.141579137Z"}
{"log":"Traceback (most recent call last):\n","stream":"stderr","time":"2022-04-04T06:50:17.141582877Z"}
{"log":"  File \"/usr/src/app/chaos_genius/controllers/data_source_metadata_controller.py\", line 146, in run_metadata_prefetch\n","stream":"stderr","time":"2022-04-04T06:50:17.141586651Z"}
{"log":"    _ = scan_db_and_save_table_info(\n","stream":"stderr","time":"2022-04-04T06:50:17.141590788Z"}
{"log":"  File \"/usr/src/app/chaos_genius/controllers/data_source_metadata_controller.py\", line 208, in scan_db_and_save_table_info\n","stream":"stderr","time":"2022-04-04T06:50:17.14159453Z"}
{"log":"    table_info = get_table_info(\n","stream":"stderr","time":"2022-04-04T06:50:17.141598702Z"}
{"log":"  File \"/usr/src/app/chaos_genius/connectors/__init__.py\", line 80, in get_table_info\n","stream":"stderr","time":"2022-04-04T06:50:17.141602506Z"}
{"log":"    table_info[\"columns\"] = db_connection.get_columns(table_name, schema)\n","stream":"stderr","time":"2022-04-04T06:50:17.141606394Z"}
{"log":"  File \"/usr/src/app/chaos_genius/connectors/base_db.py\", line 114, in get_columns\n","stream":"stderr","time":"2022-04-04T06:50:17.141610367Z"}
{"log":"    db_columns = self.inspector.get_columns(table_name=use_table, schema=use_schema)\n","stream":"stderr","time":"2022-04-04T06:50:17.14161443Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/reflection.py\", line 497, in get_columns\n","stream":"stderr","time":"2022-04-04T06:50:17.141618523Z"}
{"log":"    col_defs = self.dialect.get_columns(\n","stream":"stderr","time":"2022-04-04T06:50:17.141629696Z"}
{"log":"  File \"\u003cstring\u003e\", line 2, in get_columns\n","stream":"stderr","time":"2022-04-04T06:50:17.141633527Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/reflection.py\", line 55, in cache\n","stream":"stderr","time":"2022-04-04T06:50:17.141637588Z"}
{"log":"    ret = fn(self, con, *args, **kw)\n","stream":"stderr","time":"2022-04-04T06:50:17.141641448Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/dialects/postgresql/base.py\", line 3652, in get_columns\n","stream":"stderr","time":"2022-04-04T06:50:17.141645024Z"}
{"log":"    domains = self._load_domains(connection)\n","stream":"stderr","time":"2022-04-04T06:50:17.141648894Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/dialects/postgresql/base.py\", line 4420, in _load_domains\n","stream":"stderr","time":"2022-04-04T06:50:17.14165258Z"}
{"log":"    c = connection.execution_options(future_result=True).execute(s)\n","stream":"stderr","time":"2022-04-04T06:50:17.141656908Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\", line 1262, in execute\n","stream":"stderr","time":"2022-04-04T06:50:17.14166099Z"}
{"log":"    return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)\n","stream":"stderr","time":"2022-04-04T06:50:17.141665103Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py\", line 329, in _execute_on_connection\n","stream":"stderr","time":"2022-04-04T06:50:17.141669292Z"}
{"log":"    return connection._execute_clauseelement(\n","stream":"stderr","time":"2022-04-04T06:50:17.141673477Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\", line 1451, in _execute_clauseelement\n","stream":"stderr","time":"2022-04-04T06:50:17.141677314Z"}
{"log":"    ret = self._execute_context(\n","stream":"stderr","time":"2022-04-04T06:50:17.141681283Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\", line 1813, in _execute_context\n","stream":"stderr","time":"2022-04-04T06:50:17.141685087Z"}
{"log":"    self._handle_dbapi_exception(\n","stream":"stderr","time":"2022-04-04T06:50:17.14168916Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\", line 1994, in _handle_dbapi_exception\n","stream":"stderr","time":"2022-04-04T06:50:17.141692846Z"}
{"log":"    util.raise_(\n","stream":"stderr","time":"2022-04-04T06:50:17.141697078Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py\", line 211, in raise_\n","stream":"stderr","time":"2022-04-04T06:50:17.141700527Z"}
{"log":"    raise exception\n","stream":"stderr","time":"2022-04-04T06:50:17.14170438Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\", line 1770, in _execute_context\n","stream":"stderr","time":"2022-04-04T06:50:17.14170792Z"}
{"log":"    self.dialect.do_execute(\n","stream":"stderr","time":"2022-04-04T06:50:17.141712173Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py\", line 717, in do_execute\n","stream":"stderr","time":"2022-04-04T06:50:17.141716237Z"}
{"log":"    cursor.execute(statement, parameters)\n","stream":"stderr","time":"2022-04-04T06:50:17.141720096Z"}
{"log":"sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) SSL SYSCALL error: EOF detected\n","stream":"stderr","time":"2022-04-04T06:50:17.141723897Z"}
{"log":"\n","stream":"stderr","time":"2022-04-04T06:50:17.14172794Z"}
{"log":"[SQL: \n","stream":"stderr","time":"2022-04-04T06:50:17.141731442Z"}
{"log":"            SELECT t.typname as \"name\",\n","stream":"stderr","time":"2022-04-04T06:50:17.141735092Z"}
{"log":"               pg_catalog.format_type(t.typbasetype, t.typtypmod) as \"attype\",\n","stream":"stderr","time":"2022-04-04T06:50:17.141738919Z"}
{"log":"               not t.typnotnull as \"nullable\",\n","stream":"stderr","time":"2022-04-04T06:50:17.141743131Z"}
{"log":"               t.typdefault as \"default\",\n","stream":"stderr","time":"2022-04-04T06:50:17.141751647Z"}
{"log":"               pg_catalog.pg_type_is_visible(t.oid) as \"visible\",\n","stream":"stderr","time":"2022-04-04T06:50:17.141755847Z"}
{"log":"               n.nspname as \"schema\"\n","stream":"stderr","time":"2022-04-04T06:50:17.141759879Z"}
{"log":"            FROM pg_catalog.pg_type t\n","stream":"stderr","time":"2022-04-04T06:50:17.14176358Z"}
{"log":"               LEFT JOIN pg_catalog.pg_namespace n ON n.oid = t.typnamespace\n","stream":"stderr","time":"2022-04-04T06:50:17.141767302Z"}
{"log":"            WHERE t.typtype = 'd'\n","stream":"stderr","time":"2022-04-04T06:50:17.141771547Z"}
{"log":"        ]\n","stream":"stderr","time":"2022-04-04T06:50:17.141775504Z"}
{"log":"(Background on this error at: http://sqlalche.me/e/14/e3q8)\n","stream":"stderr","time":"2022-04-04T06:50:17.141779035Z"}
{"log":"[2022-04-04 06:50:17,145: INFO/ForkPoolWorker-2] Task chaos_genius.jobs.metadata_prefetch.fetch_data_source_schema[c22d705d-20dc-4b43-9d1a-39671670cfe8] succeeded in 66375.81781294598s: False\n","stream":"stderr","time":"2022-04-04T06:50:17.146113272Z"}
{"log":"[2022-04-04 06:50:43,925: INFO/MainProcess] Task chaos_genius.jobs.alert_tasks.alert_digest_daily_scheduler[a7530551-fe8c-4f26-971e-32a4ed8bd942] received\n","stream":"stderr","time":"2022-04-04T06:50:43.926092968Z"}
{"log":"[2022-04-04 06:50:43,928: INFO/ForkPoolWorker-1] Task chaos_genius.jobs.alert_tasks.alert_digest_daily_scheduler[a7530551-fe8c-4f26-971e-32a4ed8bd942] succeeded in 0.0022775530233047903s: None\n","stream":"stderr","time":"2022-04-04T06:50:43.929034972Z"}
  • the database system is in recovery mode
{"log":"{\"asctime\": \"2022-04-05 02:33:59,754\", \"levelname\": \"ERROR\", \"name\": \"chaos_genius.controllers.data_source_metadata_controller\", \"message\": \"Error in metadata prefetch.\", \"lineno\": 160, \"funcName\": \"run_metadata_prefetch\", \"filename\": \"data_source_metadata_controller.py\", \"exc_info\": \"Traceback (most recent call last):\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\\\", line 3212, in _wrap_pool_connect\\n    return fn()\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\\\", line 301, in connect\\n    return _ConnectionFairy._checkout(self)\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\\\", line 761, in _checkout\\n    fairy = _ConnectionRecord.checkout(pool)\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\\\", line 419, in checkout\\n    rec = pool._do_get()\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/impl.py\\\", line 145, in _do_get\\n    self._dec_overflow()\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py\\\", line 70, in __exit__\\n    compat.raise_(\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py\\\", line 211, in raise_\\n    raise exception\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/impl.py\\\", line 142, in _do_get\\n    return self._create_connection()\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\\\", line 247, in _create_connection\\n    return _ConnectionRecord(self)\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\\\", line 362, in __init__\\n    self.__connect()\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\\\", line 605, in __connect\\n    pool.logger.debug(\\\"Error on connect(): %s\\\", e)\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py\\\", line 70, in __exit__\\n    compat.raise_(\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py\\\", line 211, in raise_\\n    raise exception\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\\\", line 599, in __connect\\n    connection = pool._invoke_creator(self)\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/create.py\\\", line 578, in connect\\n    return dialect.connect(*cargs, **cparams)\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py\\\", line 584, in connect\\n    return self.dbapi.connect(*cargs, **cparams)\\n  File \\\"/usr/local/lib/python3.8/dist-packages/psycopg2/__init__.py\\\", line 127, in connect\\n    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)\\npsycopg2.OperationalError: FATAL:  the database system is in recovery mode\\nFATAL:  the database system is in recovery mode\\n\\n\\nThe above exception was the direct cause of the following exception:\\n\\nTraceback (most recent call last):\\n  File \\\"/usr/src/app/chaos_genius/controllers/data_source_metadata_controller.py\\\", line 138, in run_metadata_prefetch\\n    schema_list, old_schemas_list = scan_db_and_save_schema_list(\\n  File \\\"/usr/src/app/chaos_genius/controllers/data_source_metadata_controller.py\\\", line 172, in scan_db_and_save_schema_list\\n    schema_list = get_schema_names({}, from_db_conn=True, db_conn=db_connection)\\n  File \\\"/usr/src/app/chaos_genius/connectors/__init__.py\\\", line 93, in get_schema_names\\n    db_connection.init_inspector()\\n  File \\\"/usr/src/app/chaos_genius/connectors/base_db.py\\\", line 33, in init_inspector\\n    self.inspector = SQLAlchemy().inspect(self.engine)\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/inspection.py\\\", line 64, in inspect\\n    ret = reg(subject)\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/reflection.py\\\", line 182, in _engine_insp\\n    return Inspector._construct(Inspector._init_engine, bind)\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/reflection.py\\\", line 117, in _construct\\n    init(self, bind)\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/reflection.py\\\", line 128, in _init_engine\\n    engine.connect().close()\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\\\", line 3166, in connect\\n    return self._connection_cls(self, close_with_result=close_with_result)\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\\\", line 96, in __init__\\n    else engine.raw_connection()\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\\\", line 3245, in raw_connection\\n    return self._wrap_pool_connect(self.pool.connect, _connection)\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\\\", line 3215, in _wrap_pool_connect\\n    Connection._handle_dbapi_exception_noconnection(\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\\\", line 2068, in _handle_dbapi_exception_noconnection\\n    util.raise_(\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py\\\", line 211, in raise_\\n    raise exception\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\\\", line 3212, in _wrap_pool_connect\\n    return fn()\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\\\", line 301, in connect\\n    return _ConnectionFairy._checkout(self)\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\\\", line 761, in _checkout\\n    fairy = _ConnectionRecord.checkout(pool)\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\\\", line 419, in checkout\\n    rec = pool._do_get()\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/impl.py\\\", line 145, in _do_get\\n    self._dec_overflow()\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py\\\", line 70, in __exit__\\n    compat.raise_(\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py\\\", line 211, in raise_\\n    raise exception\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/impl.py\\\", line 142, in _do_get\\n    return self._create_connection()\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\\\", line 247, in _create_connection\\n    return _ConnectionRecord(self)\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\\\", line 362, in __init__\\n    self.__connect()\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\\\", line 605, in __connect\\n    pool.logger.debug(\\\"Error on connect(): %s\\\", e)\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py\\\", line 70, in __exit__\\n    compat.raise_(\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py\\\", line 211, in raise_\\n    raise exception\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\\\", line 599, in __connect\\n    connection = pool._invoke_creator(self)\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/create.py\\\", line 578, in connect\\n    return dialect.connect(*cargs, **cparams)\\n  File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py\\\", line 584, in connect\\n    return self.dbapi.connect(*cargs, **cparams)\\n  File \\\"/usr/local/lib/python3.8/dist-packages/psycopg2/__init__.py\\\", line 127, in connect\\n    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)\\nsqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL:  the database system is in recovery mode\\nFATAL:  the database system is in recovery mode\\n\\n(Background on this error at: http://sqlalche.me/e/14/e3q8)\"}\n","stream":"stderr","time":"2022-04-05T02:33:59.757577703Z"}
{"log":"[2022-04-05 02:33:59,754: ERROR/ForkPoolWorker-1] Error in metadata prefetch.\n","stream":"stderr","time":"2022-04-05T02:33:59.757919714Z"}
{"log":"Traceback (most recent call last):\n","stream":"stderr","time":"2022-04-05T02:33:59.757927652Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\", line 3212, in _wrap_pool_connect\n","stream":"stderr","time":"2022-04-05T02:33:59.757930992Z"}
{"log":"    return fn()\n","stream":"stderr","time":"2022-04-05T02:33:59.757934198Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\", line 301, in connect\n","stream":"stderr","time":"2022-04-05T02:33:59.757937062Z"}
{"log":"    return _ConnectionFairy._checkout(self)\n","stream":"stderr","time":"2022-04-05T02:33:59.757940085Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\", line 761, in _checkout\n","stream":"stderr","time":"2022-04-05T02:33:59.757942904Z"}
{"log":"    fairy = _ConnectionRecord.checkout(pool)\n","stream":"stderr","time":"2022-04-05T02:33:59.757946236Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\", line 419, in checkout\n","stream":"stderr","time":"2022-04-05T02:33:59.75794905Z"}
{"log":"    rec = pool._do_get()\n","stream":"stderr","time":"2022-04-05T02:33:59.757952031Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/impl.py\", line 145, in _do_get\n","stream":"stderr","time":"2022-04-05T02:33:59.757955043Z"}
{"log":"    self._dec_overflow()\n","stream":"stderr","time":"2022-04-05T02:33:59.757964155Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py\", line 70, in __exit__\n","stream":"stderr","time":"2022-04-05T02:33:59.757968649Z"}
{"log":"    compat.raise_(\n","stream":"stderr","time":"2022-04-05T02:33:59.757973145Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py\", line 211, in raise_\n","stream":"stderr","time":"2022-04-05T02:33:59.757976983Z"}
{"log":"    raise exception\n","stream":"stderr","time":"2022-04-05T02:33:59.757980365Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/impl.py\", line 142, in _do_get\n","stream":"stderr","time":"2022-04-05T02:33:59.757983162Z"}
{"log":"    return self._create_connection()\n","stream":"stderr","time":"2022-04-05T02:33:59.757986062Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\", line 247, in _create_connection\n","stream":"stderr","time":"2022-04-05T02:33:59.757988908Z"}
{"log":"    return _ConnectionRecord(self)\n","stream":"stderr","time":"2022-04-05T02:33:59.75799222Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\", line 362, in __init__\n","stream":"stderr","time":"2022-04-05T02:33:59.757995013Z"}
{"log":"    self.__connect()\n","stream":"stderr","time":"2022-04-05T02:33:59.757997898Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\", line 605, in __connect\n","stream":"stderr","time":"2022-04-05T02:33:59.758000692Z"}
{"log":"    pool.logger.debug(\"Error on connect(): %s\", e)\n","stream":"stderr","time":"2022-04-05T02:33:59.758003723Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py\", line 70, in __exit__\n","stream":"stderr","time":"2022-04-05T02:33:59.758006669Z"}
{"log":"    compat.raise_(\n","stream":"stderr","time":"2022-04-05T02:33:59.758009598Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py\", line 211, in raise_\n","stream":"stderr","time":"2022-04-05T02:33:59.758012327Z"}
{"log":"    raise exception\n","stream":"stderr","time":"2022-04-05T02:33:59.758015338Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\", line 599, in __connect\n","stream":"stderr","time":"2022-04-05T02:33:59.758018102Z"}
{"log":"    connection = pool._invoke_creator(self)\n","stream":"stderr","time":"2022-04-05T02:33:59.758021086Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/create.py\", line 578, in connect\n","stream":"stderr","time":"2022-04-05T02:33:59.758023902Z"}
{"log":"    return dialect.connect(*cargs, **cparams)\n","stream":"stderr","time":"2022-04-05T02:33:59.75802687Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py\", line 584, in connect\n","stream":"stderr","time":"2022-04-05T02:33:59.758030112Z"}
{"log":"    return self.dbapi.connect(*cargs, **cparams)\n","stream":"stderr","time":"2022-04-05T02:33:59.758033124Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/psycopg2/__init__.py\", line 127, in connect\n","stream":"stderr","time":"2022-04-05T02:33:59.75803641Z"}
{"log":"    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)\n","stream":"stderr","time":"2022-04-05T02:33:59.758040866Z"}
{"log":"psycopg2.OperationalError: FATAL:  the database system is in recovery mode\n","stream":"stderr","time":"2022-04-05T02:33:59.758045144Z"}
{"log":"FATAL:  the database system is in recovery mode\n","stream":"stderr","time":"2022-04-05T02:33:59.758049064Z"}
{"log":"\n","stream":"stderr","time":"2022-04-05T02:33:59.758052993Z"}
{"log":"\n","stream":"stderr","time":"2022-04-05T02:33:59.758055816Z"}
{"log":"The above exception was the direct cause of the following exception:\n","stream":"stderr","time":"2022-04-05T02:33:59.758058511Z"}
{"log":"\n","stream":"stderr","time":"2022-04-05T02:33:59.75806138Z"}
{"log":"Traceback (most recent call last):\n","stream":"stderr","time":"2022-04-05T02:33:59.758064094Z"}
{"log":"  File \"/usr/src/app/chaos_genius/controllers/data_source_metadata_controller.py\", line 138, in run_metadata_prefetch\n","stream":"stderr","time":"2022-04-05T02:33:59.758070206Z"}
{"log":"    schema_list, old_schemas_list = scan_db_and_save_schema_list(\n","stream":"stderr","time":"2022-04-05T02:33:59.758073296Z"}
{"log":"  File \"/usr/src/app/chaos_genius/controllers/data_source_metadata_controller.py\", line 172, in scan_db_and_save_schema_list\n","stream":"stderr","time":"2022-04-05T02:33:59.75807616Z"}
{"log":"    schema_list = get_schema_names({}, from_db_conn=True, db_conn=db_connection)\n","stream":"stderr","time":"2022-04-05T02:33:59.758079133Z"}
{"log":"  File \"/usr/src/app/chaos_genius/connectors/__init__.py\", line 93, in get_schema_names\n","stream":"stderr","time":"2022-04-05T02:33:59.758081984Z"}
{"log":"    db_connection.init_inspector()\n","stream":"stderr","time":"2022-04-05T02:33:59.758084951Z"}
{"log":"  File \"/usr/src/app/chaos_genius/connectors/base_db.py\", line 33, in init_inspector\n","stream":"stderr","time":"2022-04-05T02:33:59.758087759Z"}
{"log":"    self.inspector = SQLAlchemy().inspect(self.engine)\n","stream":"stderr","time":"2022-04-05T02:33:59.758090691Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/inspection.py\", line 64, in inspect\n","stream":"stderr","time":"2022-04-05T02:33:59.758093484Z"}
{"log":"    ret = reg(subject)\n","stream":"stderr","time":"2022-04-05T02:33:59.758096375Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/reflection.py\", line 182, in _engine_insp\n","stream":"stderr","time":"2022-04-05T02:33:59.758099127Z"}
{"log":"    return Inspector._construct(Inspector._init_engine, bind)\n","stream":"stderr","time":"2022-04-05T02:33:59.758102067Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/reflection.py\", line 117, in _construct\n","stream":"stderr","time":"2022-04-05T02:33:59.758104928Z"}
{"log":"    init(self, bind)\n","stream":"stderr","time":"2022-04-05T02:33:59.75810786Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/reflection.py\", line 128, in _init_engine\n","stream":"stderr","time":"2022-04-05T02:33:59.758110619Z"}
{"log":"    engine.connect().close()\n","stream":"stderr","time":"2022-04-05T02:33:59.758113545Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\", line 3166, in connect\n","stream":"stderr","time":"2022-04-05T02:33:59.758117612Z"}
{"log":"    return self._connection_cls(self, close_with_result=close_with_result)\n","stream":"stderr","time":"2022-04-05T02:33:59.758121796Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\", line 96, in __init__\n","stream":"stderr","time":"2022-04-05T02:33:59.758126164Z"}
{"log":"    else engine.raw_connection()\n","stream":"stderr","time":"2022-04-05T02:33:59.758130367Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\", line 3245, in raw_connection\n","stream":"stderr","time":"2022-04-05T02:33:59.758133728Z"}
{"log":"    return self._wrap_pool_connect(self.pool.connect, _connection)\n","stream":"stderr","time":"2022-04-05T02:33:59.75813682Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\", line 3215, in _wrap_pool_connect\n","stream":"stderr","time":"2022-04-05T02:33:59.75813971Z"}
{"log":"    Connection._handle_dbapi_exception_noconnection(\n","stream":"stderr","time":"2022-04-05T02:33:59.758142747Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\", line 2068, in _handle_dbapi_exception_noconnection\n","stream":"stderr","time":"2022-04-05T02:33:59.758145573Z"}
{"log":"    util.raise_(\n","stream":"stderr","time":"2022-04-05T02:33:59.75814856Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py\", line 211, in raise_\n","stream":"stderr","time":"2022-04-05T02:33:59.758151349Z"}
{"log":"    raise exception\n","stream":"stderr","time":"2022-04-05T02:33:59.758154354Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\", line 3212, in _wrap_pool_connect\n","stream":"stderr","time":"2022-04-05T02:33:59.758159798Z"}
{"log":"    return fn()\n","stream":"stderr","time":"2022-04-05T02:33:59.758162761Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\", line 301, in connect\n","stream":"stderr","time":"2022-04-05T02:33:59.75816554Z"}
{"log":"    return _ConnectionFairy._checkout(self)\n","stream":"stderr","time":"2022-04-05T02:33:59.758168467Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\", line 761, in _checkout\n","stream":"stderr","time":"2022-04-05T02:33:59.758171275Z"}
{"log":"    fairy = _ConnectionRecord.checkout(pool)\n","stream":"stderr","time":"2022-04-05T02:33:59.758174324Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\", line 419, in checkout\n","stream":"stderr","time":"2022-04-05T02:33:59.758177091Z"}
{"log":"    rec = pool._do_get()\n","stream":"stderr","time":"2022-04-05T02:33:59.758179984Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/impl.py\", line 145, in _do_get\n","stream":"stderr","time":"2022-04-05T02:33:59.758182878Z"}
{"log":"    self._dec_overflow()\n","stream":"stderr","time":"2022-04-05T02:33:59.758185998Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py\", line 70, in __exit__\n","stream":"stderr","time":"2022-04-05T02:33:59.75818876Z"}
{"log":"    compat.raise_(\n","stream":"stderr","time":"2022-04-05T02:33:59.758191848Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py\", line 211, in raise_\n","stream":"stderr","time":"2022-04-05T02:33:59.758195902Z"}
{"log":"    raise exception\n","stream":"stderr","time":"2022-04-05T02:33:59.758200105Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/impl.py\", line 142, in _do_get\n","stream":"stderr","time":"2022-04-05T02:33:59.758204233Z"}
{"log":"    return self._create_connection()\n","stream":"stderr","time":"2022-04-05T02:33:59.758208351Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\", line 247, in _create_connection\n","stream":"stderr","time":"2022-04-05T02:33:59.758211184Z"}
{"log":"    return _ConnectionRecord(self)\n","stream":"stderr","time":"2022-04-05T02:33:59.758214101Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\", line 362, in __init__\n","stream":"stderr","time":"2022-04-05T02:33:59.758216897Z"}
{"log":"    self.__connect()\n","stream":"stderr","time":"2022-04-05T02:33:59.758219851Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\", line 605, in __connect\n","stream":"stderr","time":"2022-04-05T02:33:59.758222595Z"}
{"log":"    pool.logger.debug(\"Error on connect(): %s\", e)\n","stream":"stderr","time":"2022-04-05T02:33:59.758225505Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py\", line 70, in __exit__\n","stream":"stderr","time":"2022-04-05T02:33:59.758228674Z"}
{"log":"    compat.raise_(\n","stream":"stderr","time":"2022-04-05T02:33:59.758231678Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py\", line 211, in raise_\n","stream":"stderr","time":"2022-04-05T02:33:59.758234418Z"}
{"log":"    raise exception\n","stream":"stderr","time":"2022-04-05T02:33:59.758237351Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\", line 599, in __connect\n","stream":"stderr","time":"2022-04-05T02:33:59.758240095Z"}
{"log":"    connection = pool._invoke_creator(self)\n","stream":"stderr","time":"2022-04-05T02:33:59.758243021Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/create.py\", line 578, in connect\n","stream":"stderr","time":"2022-04-05T02:33:59.758245851Z"}
{"log":"    return dialect.connect(*cargs, **cparams)\n","stream":"stderr","time":"2022-04-05T02:33:59.758248816Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py\", line 584, in connect\n","stream":"stderr","time":"2022-04-05T02:33:59.758255426Z"}
{"log":"    return self.dbapi.connect(*cargs, **cparams)\n","stream":"stderr","time":"2022-04-05T02:33:59.758258416Z"}
{"log":"  File \"/usr/local/lib/python3.8/dist-packages/psycopg2/__init__.py\", line 127, in connect\n","stream":"stderr","time":"2022-04-05T02:33:59.75826123Z"}
{"log":"    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)\n","stream":"stderr","time":"2022-04-05T02:33:59.75826422Z"}
{"log":"sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL:  the database system is in recovery mode\n","stream":"stderr","time":"2022-04-05T02:33:59.75826708Z"}
{"log":"FATAL:  the database system is in recovery mode\n","stream":"stderr","time":"2022-04-05T02:33:59.758270332Z"}
{"log":"\n","stream":"stderr","time":"2022-04-05T02:33:59.758274442Z"}
{"log":"(Background on this error at: http://sqlalche.me/e/14/e3q8)\n","stream":"stderr","time":"2022-04-05T02:33:59.758278241Z"}
{"log":"[2022-04-05 02:33:59,760: INFO/ForkPoolWorker-1] Task chaos_genius.jobs.metadata_prefetch.fetch_data_source_schema[4ceeb7a0-aaa8-40c5-8a08-972f2a1a5ae0] succeeded in 0.035913983010686934s: False\n","stream":"stderr","time":"2022-04-05T02:33:59.760373818Z"}
@rjdp rjdp added the 🐛 bug Something isn't working label Apr 7, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
🐛 bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@rjdp