Unable to run Kedro in Azure with SQLiteSessionStore #1991

Dekermanjian · 2024-01-23T17:24:00Z

Description

Whenever I try to load the catalog either using a jupyter lab or IPython interface I get the following error:
Failed to instantiate session store of type 'kedro_viz.integrations.kedro.sqlite_store.SQLiteStore'

Context

I am not able to set up a new project on the cloud (Azure compute instance)

Steps to Reproduce

1). Create a new kedro project on the cloud
2). Run kedro ipython

Expected Result

The catalog/context should load like it normally does in local development

Actual Result

An error message is thrown about a locked SQLAlchemy database with the following error:
Failed to instantiate session store of type 'kedro_viz.integrations.kedro.sqlite_store.SQLiteStore'

Failed to instantiate session store of type 'kedro_viz.integrations.kedro.sqlite_store.SQLiteStore'

Your Environment

Brand new conda env with fresh installation of kedro

Kedro version used (pip show kedro or kedro -V): 0.19.2
Python version used (python -V): 3.10.13
Operating system and version: Cloud compute running linux

The text was updated successfully, but these errors were encountered:

noklam · 2024-01-23T17:34:33Z

@Dekermanjian Thanks for reporting the issue, can you show the full stacktrace and logs if possible? Since you mention:

An error message is thrown about a locked SQLAlchemy database with the following error:

Does it works with any kedro command ?

Dekermanjian · 2024-01-23T17:50:17Z

Thank you for the quick response @noklam
Yes, of course here is the fill stacktrace plus some logs:

Python 3.10.13 (main, Sep 11 2023, 13:44:35) [GCC 11.2.0]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.20.0 -- An enhanced Interactive Python. Type '?' for help.
[01/23/24 17:48:40] INFO     Resolved project path as:                                                                                      __init__.py:149
                             /mnt/batch/tasks/shared/LS_root/mounts/clusters/compute-instance-jd2/code/Users/jdekermanjian/Projects/monocle                
                             -analytics.                                                                                                                   
                             To set a different path, run '%reload_kedro <project_root>'                                                                   
[TerminalIPythonApp] WARNING | Error in loading extension: kedro.ipython
Check your config files in /home/azureuser/.ipython/profile_default
Traceback (most recent call last):
  File "/anaconda/envs/monocle_analytics/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1969, in _exec_single_context
    self.dialect.do_execute(
  File "/anaconda/envs/monocle_analytics/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 922, in do_execute
    cursor.execute(statement, parameters)
sqlite3.OperationalError: database is locked

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/anaconda/envs/monocle_analytics/lib/python3.10/site-packages/kedro/framework/session/session.py", line 195, in _init_store
    return store_class(**store_args)  # type: ignore[no-any-return]
  File "/anaconda/envs/monocle_analytics/lib/python3.10/site-packages/kedro_viz/integrations/kedro/sqlite_store.py", line 42, in __init__
    self._db_session_class = make_db_session_factory(self.location)
  File "/anaconda/envs/monocle_analytics/lib/python3.10/site-packages/kedro_viz/database.py", line 17, in make_db_session_factory
    Base.metadata.create_all(bind=engine)
  File "/anaconda/envs/monocle_analytics/lib/python3.10/site-packages/sqlalchemy/sql/schema.py", line 5832, in create_all
    bind._run_ddl_visitor(
  File "/anaconda/envs/monocle_analytics/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3244, in _run_ddl_visitor
    conn._run_ddl_visitor(visitorcallable, element, **kwargs)
  File "/anaconda/envs/monocle_analytics/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2448, in _run_ddl_visitor
    visitorcallable(self.dialect, self, **kwargs).traverse_single(element)
  File "/anaconda/envs/monocle_analytics/lib/python3.10/site-packages/sqlalchemy/sql/visitors.py", line 671, in traverse_single
    return meth(obj, **kw)
  File "/anaconda/envs/monocle_analytics/lib/python3.10/site-packages/sqlalchemy/sql/ddl.py", line 919, in visit_metadata
    self.traverse_single(
  File "/anaconda/envs/monocle_analytics/lib/python3.10/site-packages/sqlalchemy/sql/visitors.py", line 671, in traverse_single
    return meth(obj, **kw)
  File "/anaconda/envs/monocle_analytics/lib/python3.10/site-packages/sqlalchemy/sql/ddl.py", line 957, in visit_table
    )._invoke_with(self.connection)
  File "/anaconda/envs/monocle_analytics/lib/python3.10/site-packages/sqlalchemy/sql/ddl.py", line 315, in _invoke_with
    return bind.execute(self)
  File "/anaconda/envs/monocle_analytics/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1416, in execute
    return meth(
  File "/anaconda/envs/monocle_analytics/lib/python3.10/site-packages/sqlalchemy/sql/ddl.py", line 181, in _execute_on_connection
    return connection._execute_ddl(
  File "/anaconda/envs/monocle_analytics/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1528, in _execute_ddl
    ret = self._execute_context(
  File "/anaconda/envs/monocle_analytics/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1848, in _execute_context
    return self._exec_single_context(
  File "/anaconda/envs/monocle_analytics/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1988, in _exec_single_context
    self._handle_dbapi_exception(
  File "/anaconda/envs/monocle_analytics/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2344, in _handle_dbapi_exception
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
  File "/anaconda/envs/monocle_analytics/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1969, in _exec_single_context
    self.dialect.do_execute(
  File "/anaconda/envs/monocle_analytics/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 922, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked
[SQL: 
CREATE TABLE runs (
        id VARCHAR NOT NULL, 
        blob JSON, 
        PRIMARY KEY (id)
)

]
(Background on this error at: https://sqlalche.me/e/20/e3q8)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/anaconda/envs/monocle_analytics/lib/python3.10/site-packages/IPython/core/shellapp.py", line 282, in init_extensions
    self.shell.extension_manager.load_extension(ext)
  File "/anaconda/envs/monocle_analytics/lib/python3.10/site-packages/IPython/core/extensions.py", line 76, in load_extension
    return self._load_extension(module_str)
  File "/anaconda/envs/monocle_analytics/lib/python3.10/site-packages/IPython/core/extensions.py", line 93, in _load_extension
    if self._call_load_ipython_extension(mod):
  File "/anaconda/envs/monocle_analytics/lib/python3.10/site-packages/IPython/core/extensions.py", line 145, in _call_load_ipython_extension
    mod.load_ipython_extension(self.shell)
  File "/anaconda/envs/monocle_analytics/lib/python3.10/site-packages/kedro/ipython/__init__.py", line 47, in load_ipython_extension
    reload_kedro()
  File "/anaconda/envs/monocle_analytics/lib/python3.10/site-packages/kedro/ipython/__init__.py", line 101, in reload_kedro
    session = KedroSession.create(
  File "/anaconda/envs/monocle_analytics/lib/python3.10/site-packages/kedro/framework/session/session.py", line 150, in create
    session = cls(
  File "/anaconda/envs/monocle_analytics/lib/python3.10/site-packages/kedro/framework/session/session.py", line 111, in __init__
    self._store = self._init_store()
  File "/anaconda/envs/monocle_analytics/lib/python3.10/site-packages/kedro/framework/session/session.py", line 202, in _init_store
    raise ValueError(
ValueError: 
(sqlite3.OperationalError) database is locked
[SQL: 
CREATE TABLE runs (
        id VARCHAR NOT NULL, 
        blob JSON, 
        PRIMARY KEY (id)
)

]
(Background on this error at: https://sqlalche.me/e/20/e3q8).
Failed to instantiate session store of type 'kedro_viz.integrations.kedro.sqlite_store.SQLiteStore'.

noklam · 2024-01-23T17:59:26Z

@Dekermanjian Are you able to run the same project outside of a VM environment?

Something that caught my attention is the error shows two very different paths. The error itself is suggesting some locking issues that multi-process/thread is trying to access it at the same time.

[01/23/24 17:48:40] INFO Resolved project path as: init.py:149
/mnt/batch/tasks/shared/LS_root/mounts/clusters/compute-instance-jd2/code/Users/jdekermanjian/Projects/monocle
-analytics.
Traceback (most recent call last):
File "/anaconda/envs/monocle_analytics/lib/python3.10/site-packages/IPython/core/shellapp.py",

Dekermanjian · 2024-01-23T18:19:52Z

@noklam Thank you for helping me with this, I really appreciate it.
I literally just created a new kedro project using the kedro new command. There is nothing in my project yet. I wanted to run an .ipynb notebook and start developing but the catalog would not load. Usually, I can get catalog.list() to print out and empty list when I first initialize a project.

Dekermanjian · 2024-01-23T18:45:25Z

I can confirm that when I don't use kedro-viz plugin it works as expected.

noklam · 2024-01-23T22:23:52Z

@Dekermanjian that's quite weird. We have CI job running in Linux regularly,I don't know what's different in a cloud VM.

Which version of Kedro-viz are you using? If you just clone the exact same project to a local env would it work?

Dekermanjian · 2024-01-23T22:28:33Z

@noklam
Yeah, I am not sure why this is happening either. I am running this on a managed azure compute instance in Azure ML https://learn.microsoft.com/en-us/azure/machine-learning/concept-compute-instance?view=azureml-api-2 . Locally, I have no issues with kedro-viz.

miranda-datarisk · 2024-02-20T14:00:54Z

Same here. My project was moved to Azure ML yesterday and I can't run anything.
I did the same test starting a project from scratch, and this error appeared. Running with no problems locally.

cyrenaique · 2024-02-29T11:59:56Z

same here with version 0.19-3 (no pb with 0.18.14) on some cifs drives (linux), solved the problem by commenting in settings.py :
from kedro_viz.integrations.kedro.sqlite_store import SQLiteStore # noqa: E402
SESSION_STORE_CLASS = SQLiteStore
I guess that's not the right solution but so far....

noklam · 2024-02-29T12:19:35Z

Thanks for letting us know the issue and sorry this causes some bumpy experience. We are unable to reproduce this locally so it making this hard for us to fix quickly. I'll flag it to the team again.

cyrenaique · 2024-02-29T13:43:51Z

To reproduce , our config: ubuntu22.04 (mounted storage) /etc/fstab : /mnt/shares/L cifs credentials=/root/.cifspwd,auto,rw,user,noperm... and from a windows client on same storage some problems with shutils.copy if I remember well.
Both work perfectly if created and run on local drives.
Hope it can help

tdhooghe · 2024-05-22T09:02:46Z

Also wanted to let you know that I am facing the same issue as described above. I am using an Azure AI | Machine Learning Studio compute instance to run Kedro and have Kedro Viz installed as well.

Linux 5.15.0-1040-azure kedro-org/kedro#47~20.04.1-Ubuntu SMP Fri Jun 2 21:38:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
kedro                        0.19.5
kedro-datasets               3.0.0
kedro-telemetry              0.4.0
kedro-viz                    9.0.0

The stacktrace looks as follows:

Traceback (most recent call last):
  File "/anaconda/envs/py311/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1144, in _commit_impl
    self.engine.dialect.do_commit(self.connection)
  File "/anaconda/envs/py311/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 701, in do_commit
    dbapi_connection.commit()
sqlite3.OperationalError: database is locked

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/anaconda/envs/py311/bin/kedro", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/anaconda/envs/py311/lib/python3.11/site-packages/kedro/framework/cli/cli.py", line 233, in main
    cli_collection()
  File "/anaconda/envs/py311/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/anaconda/envs/py311/lib/python3.11/site-packages/kedro/framework/cli/cli.py", line 130, in main
    super().main(
  File "/anaconda/envs/py311/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/anaconda/envs/py311/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/anaconda/envs/py311/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/anaconda/envs/py311/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/anaconda/envs/py311/lib/python3.11/site-packages/kedro/framework/cli/project.py", line 222, in run
    with KedroSession.create(
  File "/anaconda/envs/py311/lib/python3.11/site-packages/kedro/framework/session/session.py", line 273, in __exit__
    self.close()
  File "/anaconda/envs/py311/lib/python3.11/site-packages/kedro/framework/session/session.py", line 265, in close
    self._store.save()
  File "/anaconda/envs/py311/lib/python3.11/site-packages/kedro_viz/integrations/kedro/sqlite_store.py", line 84, in save
    with self._db_session_class.begin() as session:
  File "/anaconda/envs/py311/lib/python3.11/contextlib.py", line 144, in __exit__
    next(self.gen)
  File "/anaconda/envs/py311/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 1801, in _maker_context_manager
    with self.begin():
  File "/anaconda/envs/py311/lib/python3.11/site-packages/sqlalchemy/engine/util.py", line 147, in __exit__
    with util.safe_reraise():
  File "/anaconda/envs/py311/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/anaconda/envs/py311/lib/python3.11/site-packages/sqlalchemy/engine/util.py", line 145, in __exit__
    self.commit()
  File "<string>", line 2, in commit
  File "/anaconda/envs/py311/lib/python3.11/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
    ret_value = fn(self, *arg, **kw)
                ^^^^^^^^^^^^^^^^^^^^
  File "/anaconda/envs/py311/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 1309, in commit
    trans.commit()
  File "/anaconda/envs/py311/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 2629, in commit
    self._do_commit()
  File "/anaconda/envs/py311/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 2734, in _do_commit
    self._connection_commit_impl()
  File "/anaconda/envs/py311/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 2705, in _connection_commit_impl
    self.connection._commit_impl()
  File "/anaconda/envs/py311/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1146, in _commit_impl
    self._handle_dbapi_exception(e, None, None, None, None)
  File "/anaconda/envs/py311/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 2353, in _handle_dbapi_exception
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
  File "/anaconda/envs/py311/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1144, in _commit_impl
    self.engine.dialect.do_commit(self.connection)
  File "/anaconda/envs/py311/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 701, in do_commit
    dbapi_connection.commit()
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked
(Background on this error at: https://sqlalche.me/e/20/e3q8)

noklam · 2024-07-18T14:36:14Z

I am moving this to kedro-viz, as I think the main issue seems to be Azure specific when using SQLiteSessionStore

gyasibaaye · 2024-08-12T22:48:25Z

Any updates on this issue? Commenting out the lines mentioned below does work, but that shouldn't be necessary to begin

same here with version 0.19-3 (no pb with 0.18.14) on some cifs drives (linux), solved the problem by commenting in settings.py : from kedro_viz.integrations.kedro.sqlite_store import SQLiteStore # noqa: E402 SESSION_STORE_CLASS = SQLiteStore I guess that's not the right solution but so far....

ravi-kumar-pilla · 2024-08-23T21:14:13Z

Hi ,

Sorry for the delayed response. Thanks for raising the issue. There seems to be multiple processes trying to access SQLite database that is initialized in settings.py for Kedro-Viz Experiment Tracking and SQLite allows one writer at a time.

At this moment, I do not have a permanent fix but we will take this as priority in the next sprint. For now, the work-around would be to comment out SESSION_STORE_CLASS = SQLiteStore instantiation in settings.py, if you are not using Experiment Tracking on Kedro-Viz.

Thank you

SajidAlamQB · 2024-10-14T11:54:33Z

Completed in: #2131

noklam added the Community label Jan 25, 2024

github-actions bot mentioned this issue Feb 1, 2024

Monthly issue metrics report kedro-org/kedro#3582

Open

noklam changed the title ~~Unable to run Kedro in cloud VM compute~~ Unable to run Kedro in Azure with SQLiteSessionStore Jul 18, 2024

noklam transferred this issue from kedro-org/kedro Jul 18, 2024

ravi-kumar-pilla added this to Kedro-Viz Aug 23, 2024

ravi-kumar-pilla removed this from Kedro Framework Aug 23, 2024

ravi-kumar-pilla moved this to Backlog in Kedro-Viz Aug 23, 2024

rashidakanchwala moved this from Backlog to Todo in Kedro-Viz Sep 2, 2024

rashidakanchwala assigned SajidAlamQB Sep 30, 2024

SajidAlamQB mentioned this issue Oct 9, 2024

Fix Kedro in Azure ML with SQLiteSessionStore #2131

Merged

5 tasks

SajidAlamQB closed this as completed Oct 14, 2024

github-project-automation bot moved this from In Review to Done in Kedro-Viz Oct 14, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to run Kedro in Azure with SQLiteSessionStore #1991

Unable to run Kedro in Azure with SQLiteSessionStore #1991

Dekermanjian commented Jan 23, 2024

noklam commented Jan 23, 2024 •

edited

Loading

Dekermanjian commented Jan 23, 2024 •

edited

Loading

noklam commented Jan 23, 2024

Dekermanjian commented Jan 23, 2024

Dekermanjian commented Jan 23, 2024

noklam commented Jan 23, 2024

Dekermanjian commented Jan 23, 2024

miranda-datarisk commented Feb 20, 2024

cyrenaique commented Feb 29, 2024 •

edited

Loading

noklam commented Feb 29, 2024

cyrenaique commented Feb 29, 2024

tdhooghe commented May 22, 2024 •

edited

Loading

noklam commented Jul 18, 2024

gyasibaaye commented Aug 12, 2024

ravi-kumar-pilla commented Aug 23, 2024

SajidAlamQB commented Oct 14, 2024

Unable to run Kedro in Azure with SQLiteSessionStore #1991

Unable to run Kedro in Azure with SQLiteSessionStore #1991

Comments

Dekermanjian commented Jan 23, 2024

Description

Context

Steps to Reproduce

Expected Result

Actual Result

Your Environment

noklam commented Jan 23, 2024 • edited Loading

Dekermanjian commented Jan 23, 2024 • edited Loading

noklam commented Jan 23, 2024

Dekermanjian commented Jan 23, 2024

Dekermanjian commented Jan 23, 2024

noklam commented Jan 23, 2024

Dekermanjian commented Jan 23, 2024

miranda-datarisk commented Feb 20, 2024

cyrenaique commented Feb 29, 2024 • edited Loading

noklam commented Feb 29, 2024

cyrenaique commented Feb 29, 2024

tdhooghe commented May 22, 2024 • edited Loading

noklam commented Jul 18, 2024

gyasibaaye commented Aug 12, 2024

ravi-kumar-pilla commented Aug 23, 2024

SajidAlamQB commented Oct 14, 2024

noklam commented Jan 23, 2024 •

edited

Loading

Dekermanjian commented Jan 23, 2024 •

edited

Loading

cyrenaique commented Feb 29, 2024 •

edited

Loading

tdhooghe commented May 22, 2024 •

edited

Loading