Neo4J Dataprep Microservice: Cant start with Python due to TypeError: cannot pickle '_thread.RLock' object

[ajaykallepalli]

Similar to Github Issue: #841

Full error message:

(venv) ➜  langchain git:(main) ✗ python prepare_doc_neo4j.py                        
/Users/ajaykallepalli/Documents/GitHub/GenAIComps/venv/lib/python3.12/site-packages/pydantic/_internal/_fields.py:132: UserWarning: Field "model_name_or_path" in Audio2TextDoc has conflict with protected namespace "model_".


You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
 warnings.warn(
/Users/ajaykallepalli/Documents/GitHub/GenAIComps/venv/lib/python3.12/site-packages/langchain/__init__.py:30: UserWarning: Importing LLMChain from langchain root module is no longer supported. Please use langchain.chains.LLMChain instead.
 warnings.warn(
/Users/ajaykallepalli/Documents/GitHub/GenAIComps/venv/lib/python3.12/site-packages/langchain/__init__.py:30: UserWarning: Importing PromptTemplate from langchain root module is no longer supported. Please use langchain_core.prompts.PromptTemplate instead.
 warnings.warn(
[2024-10-31 09:51:29,281] [    INFO] - Base service - CORS is enabled.
[2024-10-31 09:51:29,282] [    INFO] - Base service - Setting up HTTP server
[2024-10-31 09:51:29,283] [    INFO] - Base service - Uvicorn server setup on port 6007
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:6007 (Press CTRL+C to quit)
[2024-10-31 09:51:29,286] [    INFO] - Base service - HTTP server setup successful
Traceback (most recent call last):
 File "/Users/ajaykallepalli/Documents/GitHub/GenAIComps/comps/dataprep/neo4j/langchain/prepare_doc_neo4j.py", line 190, in <module>
   opea_microservices["opea_service@prepare_doc_neo4j"].start()
 File "/Users/ajaykallepalli/Documents/GitHub/GenAIComps/comps/cores/mega/micro_service.py", line 122, in start
   self.process.start()
 File "/opt/homebrew/anaconda3/lib/python3.12/multiprocessing/process.py", line 121, in start
   self._popen = self._Popen(self)
                 ^^^^^^^^^^^^^^^^^
 File "/opt/homebrew/anaconda3/lib/python3.12/multiprocessing/context.py", line 224, in _Popen
   return _default_context.get_context().Process._Popen(process_obj)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/opt/homebrew/anaconda3/lib/python3.12/multiprocessing/context.py", line 289, in _Popen
   return Popen(process_obj)
          ^^^^^^^^^^^^^^^^^^
 File "/opt/homebrew/anaconda3/lib/python3.12/multiprocessing/popen_spawn_posix.py", line 32, in __init__
   super().__init__(process_obj)
 File "/opt/homebrew/anaconda3/lib/python3.12/multiprocessing/popen_fork.py", line 19, in __init__
   self._launch(process_obj)
 File "/opt/homebrew/anaconda3/lib/python3.12/multiprocessing/popen_spawn_posix.py", line 47, in _launch
   reduction.dump(process_obj, fp)
 File "/opt/homebrew/anaconda3/lib/python3.12/multiprocessing/reduction.py", line 60, in dump
   ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle '_thread.RLock' object

Steps Taken:

1. Clone the repo
2. Create venv, pip install -e .
3. move to comps/dataprep/neo4j/langchain/
4. Follow readme
   • pip install requirements
   • brew install tesseract
   • pip install python-poppler
   • Docker setup for python (login and access to neo4j through local host is working)
   • Set Neo4J environment variables and pythonpath
   • Set environment variables
   • Run Python script

docker run \
    -p 7474:7474 -p 7687:7687 \
    -v $PWD/data:/data -v $PWD/plugins:/plugins \
    --name neo4j-apoc \
    -d \
    -e NEO4J_AUTH=neo4j/password \
    -e NEO4J_PLUGINS=\[\"apoc\"\]  \
    neo4j:latest

 export NEO4J_URI="bolt://localhost:7687"
export NEO4J_USERNAME="neo4j"
export NEO4J_PASSWORD="password"

python prepare_doc_neo4j.py

Initial Python Run

Initial run raised the below error and needed to be resolved by performing conda install cairo

OSError: no library called "cairo-2" was found
no library called "cairo" was found
no library called "libcairo-2" was found

Final Error Message

Caused by running python prepare_doc_neo4j.py is given at the top of this file.

Please let me know if any other information is required.

Equipment Information

MacOS: Sequoia 15.0.1
Macbook Pro M1 chip, 32 GB RAM
Python Version: Python 3.12.4

[rbrugaro]

@XuhuiRen