TimeoutError from run_cgs_models() in first 20 seconds #127

galvaniquee · 2023-03-14T12:03:43Z

galvaniquee
Mar 14, 2023

Hi guys,

I am trying to run the pbmc_multiome_tutorial and at this point -

import pickle
cistopic_obj = pickle.load(open(os.path.join(work_dir, 'scATAC/cistopic_obj.pkl'), 'rb'))
from pycisTopic.cistopic_class import *
models=run_cgs_models(cistopic_obj,
                    n_topics=[2,4,10,16,32,48],
                    n_cpu=1,
                    n_iter=500,
                    random_state=555,
                    alpha=50,
                    alpha_by_topic=True,
                    eta=0.1,
                    eta_by_topic=False,
                    save_path=None,
                    _temp_dir = os.path.join(tmp_dir + 'ray_spill'))

And here is the error from it

TimeoutError Traceback (most recent call last)
File /opt/anaconda3/envs/scenicplus/lib/python3.8/site-packages/ray/_private/node.py:293, in Node.init(self, ray_params, head, shutdown_at_exit, spawn_reaper, connect_only, default_worker)
292 try:
--> 293 ray._private.services.wait_for_node(
294 self.gcs_address,
295 self._plasma_store_socket_name,
296 )
297 except TimeoutError:

File /opt/anaconda3/envs/scenicplus/lib/python3.8/site-packages/ray/_private/services.py:436, in wait_for_node(gcs_address, node_plasma_store_socket_name, timeout)
435 time.sleep(0.1)
--> 436 raise TimeoutError("Timed out while waiting for node to startup.")

TimeoutError: Timed out while waiting for node to startup.

During handling of the above exception, another exception occurred:

Exception Traceback (most recent call last)
Cell In[60], line 4
2 cistopic_obj = pickle.load(open(os.path.join(work_dir, 'scATAC/cistopic_obj.pkl'), 'rb'))
3 from pycisTopic.cistopic_class import *
----> 4 models=run_cgs_models(cistopic_obj,
5 n_topics=[2,4,10,16,32,48],
6 n_cpu=1,
7 n_iter=500,
8 random_state=555,
9 alpha=50,
10 alpha_by_topic=True,
11 eta=0.1,
12 eta_by_topic=False,
13 save_path=None,
14 _temp_dir = os.path.join(tmp_dir + 'ray_spill'))

File /opt/anaconda3/envs/scenicplus/lib/python3.8/site-packages/pycisTopic/lda_models.py:156, in run_cgs_models(cistopic_obj, n_topics, n_cpu, n_iter, random_state, alpha, alpha_by_topic, eta, eta_by_topic, top_topics_coh, save_path, **kwargs)
154 region_names = cistopic_obj.region_names
155 cell_names = cistopic_obj.cell_names
--> 156 ray.init(num_cpus=n_cpu, **kwargs)
157 model_list = ray.get(
158 [
159 run_cgs_model.remote(
(...)
174 ]
175 )
176 ray.shutdown()

File /opt/anaconda3/envs/scenicplus/lib/python3.8/site-packages/ray/_private/client_mode_hook.py:105, in client_mode_hook..wrapper(*args, **kwargs)
103 if func.name != "init" or is_client_mode_enabled_by_default:
104 return getattr(ray, func.name)(*args, **kwargs)
--> 105 return func(*args, **kwargs)

File /opt/anaconda3/envs/scenicplus/lib/python3.8/site-packages/ray/_private/worker.py:1451, in init(address, num_cpus, num_gpus, resources, object_store_memory, local_mode, ignore_reinit_error, include_dashboard, dashboard_host, dashboard_port, job_config, configure_logging, logging_level, logging_format, log_to_driver, namespace, runtime_env, storage, **kwargs)
1409 ray_params = ray._private.parameter.RayParams(
1410 node_ip_address=node_ip_address,
1411 raylet_ip_address=raylet_ip_address,
(...)
1445 node_name=_node_name,
1446 )
1447 # Start the Ray processes. We set shutdown_at_exit=False because we
1448 # shutdown the node in the ray.shutdown call that happens in the atexit
1449 # handler. We still spawn a reaper process in case the atexit handler
1450 # isn't called.
-> 1451 _global_node = ray._private.node.Node(
1452 head=True,
1453 shutdown_at_exit=False,
1454 spawn_reaper=True,
1455 ray_params=ray_params,
1456 )
1457 else:
1458 # In this case, we are connecting to an existing cluster.
1459 if num_cpus is not None or num_gpus is not None:

File /opt/anaconda3/envs/scenicplus/lib/python3.8/site-packages/ray/_private/node.py:298, in Node.init(self, ray_params, head, shutdown_at_exit, spawn_reaper, connect_only, default_worker)
293 ray._private.services.wait_for_node(
294 self.gcs_address,
295 self._plasma_store_socket_name,
296 )
297 except TimeoutError:
--> 298 raise Exception(
299 "The current node has not been updated within 30 "
300 "seconds, this could happen because of some of "
301 "the Ray processes failed to startup."
302 )
303 node_info = ray._private.services.get_node_to_connect_for_driver(
304 self.gcs_address,
305 self._raylet_ip_address,
306 )
307 if self._ray_params.node_manager_port == 0:

Exception: The current node has not been updated within 30 seconds, this could happen because of some of the Ray processes failed to startup.

Changing the n_cpu doesn't affect the results, may I know what I should do with it?

SeppeDeWinter · 2023-03-14T12:46:47Z

SeppeDeWinter
Mar 14, 2023
Maintainer

Hi @galvaniquee

Can you make sure that _temp_dir is set to a path that exists on your disk?

Best,

Seppe

6 replies

SeppeDeWinter Mar 14, 2023
Maintainer

Another issue that often occurs is that the length of the path of the temp_dir is too long, see #24

Can you check this.

Another thing you can try is running inside a python kernel, this is what is throwing the error.

import ray
ray.init(n_cpu = 3, _temp_dir = os.path.join(tmp_dir + 'ray_spill'))

Best,

Seppe

galvaniquee Mar 14, 2023
Author

Hi Seppe,

I just tried

import ray
ray.init(num_cpus=3, _temp_dir = os.path.join(tmp_dir + 'ray_spill'))
ray.shutdown()

And same error -

---------------------------------------------------------------------------
TimeoutError                              Traceback (most recent call last)
File /opt/anaconda3/envs/scenicplus/lib/python3.8/site-packages/ray/_private/node.py:293, in Node.__init__(self, ray_params, head, shutdown_at_exit, spawn_reaper, connect_only, default_worker)
    292 try:
--> 293     ray._private.services.wait_for_node(
    294         self.gcs_address,
    295         self._plasma_store_socket_name,
    296     )
    297 except TimeoutError:

File /opt/anaconda3/envs/scenicplus/lib/python3.8/site-packages/ray/_private/services.py:436, in wait_for_node(gcs_address, node_plasma_store_socket_name, timeout)
    435         time.sleep(0.1)
--> 436 raise TimeoutError("Timed out while waiting for node to startup.")

TimeoutError: Timed out while waiting for node to startup.

During handling of the above exception, another exception occurred:

Exception                                 Traceback (most recent call last)
Cell In[68], line 2
      1 import ray
----> 2 ray.init(num_cpus=3, _temp_dir = os.path.join(tmp_dir + 'ray_spill'))
      3 ray.shutdown()

File /opt/anaconda3/envs/scenicplus/lib/python3.8/site-packages/ray/_private/client_mode_hook.py:105, in client_mode_hook.<locals>.wrapper(*args, **kwargs)
    103     if func.__name__ != "init" or is_client_mode_enabled_by_default:
    104         return getattr(ray, func.__name__)(*args, **kwargs)
--> 105 return func(*args, **kwargs)

File /opt/anaconda3/envs/scenicplus/lib/python3.8/site-packages/ray/_private/worker.py:1451, in init(address, num_cpus, num_gpus, resources, object_store_memory, local_mode, ignore_reinit_error, include_dashboard, dashboard_host, dashboard_port, job_config, configure_logging, logging_level, logging_format, log_to_driver, namespace, runtime_env, storage, **kwargs)
   1409     ray_params = ray._private.parameter.RayParams(
   1410         node_ip_address=node_ip_address,
   1411         raylet_ip_address=raylet_ip_address,
   (...)
   1445         node_name=_node_name,
   1446     )
   1447     # Start the Ray processes. We set shutdown_at_exit=False because we
   1448     # shutdown the node in the ray.shutdown call that happens in the atexit
   1449     # handler. We still spawn a reaper process in case the atexit handler
   1450     # isn't called.
-> 1451     _global_node = ray._private.node.Node(
   1452         head=True,
   1453         shutdown_at_exit=False,
   1454         spawn_reaper=True,
   1455         ray_params=ray_params,
   1456     )
   1457 else:
   1458     # In this case, we are connecting to an existing cluster.
   1459     if num_cpus is not None or num_gpus is not None:

File /opt/anaconda3/envs/scenicplus/lib/python3.8/site-packages/ray/_private/node.py:298, in Node.__init__(self, ray_params, head, shutdown_at_exit, spawn_reaper, connect_only, default_worker)
    293     ray._private.services.wait_for_node(
    294         self.gcs_address,
    295         self._plasma_store_socket_name,
    296     )
    297 except TimeoutError:
--> 298     raise Exception(
    299         "The current node has not been updated within 30 "
    300         "seconds, this could happen because of some of "
    301         "the Ray processes failed to startup."
    302     )
    303 node_info = ray._private.services.get_node_to_connect_for_driver(
    304     self.gcs_address,
    305     self._raylet_ip_address,
    306 )
    307 if self._ray_params.node_manager_port == 0:

Exception: The current node has not been updated within 30 seconds, this could happen because of some of the Ray processes failed to startup.

Do you think it's the setup of ray? I also tried without num_cpus or num_cpus=1, same error as well.

galvaniquee Mar 14, 2023
Author

Hi Seppe,

I figure out a way to run the "init" command, and it only works if I remove the _temp_dir command.

import ray
ray.init(num_cpus=1)
ray.shutdown()

Any thoughts on why this is happening?

SeppeDeWinter Mar 14, 2023
Maintainer

Hi @galvaniquee

Like I said above, it could be that the path is too long.
See this issue: #24

Could it be this?

Best,

Seppe

galvaniquee Mar 15, 2023
Author

Hi Seppe,

I don't think that's the issue as my path is very short at the moment. However, by removing the _temp_dir command it is now working as expected. Hope this will help some other users too. I have more questions regarding where can I obtain the databases in the file

db_fpath = "/staging/leuven/stg_00002/lcb/icistarget/data/make_rankings/v10_clust/CTX_hg38"
motif_annot_fpath = "/staging/leuven/stg_00002/lcb/cbravo/cluster_motif_collection_V10_no_desso_no_factorbook/snapshots"

Should I start another discussion on this or you can point me to that directly? I found this webpage - https://resources.aertslab.org/cistarget/ but not sure which one should I be fetching ...

Thanks!

SeppeDeWinter · 2023-03-15T08:18:37Z

SeppeDeWinter
Mar 15, 2023
Maintainer

It's the databases in this folder: https://resources.aertslab.org/cistarget/databases/

e.g. for human use the ones here: https://resources.aertslab.org/cistarget/databases/homo_sapiens/hg38/screen/mc_v10_clust/region_based/

I hope this helps.

Best,

Seppe

3 replies

galvaniquee Mar 20, 2023
Author

Hi Seppe,

I can tell where the two *.feathre files are, but I cannot find motifs-v10-nr.hgnc-m0.00001-o0.0.tbl in the database. Would you point me to that file? thank you

galvaniquee Mar 20, 2023
Author

Oh I just googled that - https://resources.aertslab.org/cistarget/motif2tf/
Can I confirm with you this will be the file to use? Appreciated!

SeppeDeWinter Mar 27, 2023
Maintainer

Indeed, those are the correct files.

Best,

Seppe

TingTingShao · 2024-03-11T07:47:18Z

TingTingShao
Mar 11, 2024

import ray
ray.init(num_cpus=1)
ray.shutdown()

It will give me error like this:

2024-03-11 08:45:53,351 ERROR services.py:1329 -- Failed to start the dashboard , return code -11
2024-03-11 08:45:53,352 ERROR services.py:1354 -- Error should be written to 'dashboard.log' or 'dashboard.err'. We are printing the last 20 lines for you. See 'https://docs.ray.io/en/master/ray-observability/ray-logging.html#logging-directory-structure' to find where the log file is.
2024-03-11 08:45:53,353 ERROR services.py:1398 --
The last 20 lines of /tmp/ray/session_2024-03-11_08-45-51_296170_935375/logs/dashboard.log (it contains the error message from the dashboard):
2024-03-11 08:45:53,309 INFO head.py:254 -- Starting dashboard metrics server on port 44227

2024-03-11 08:45:54,494 INFO worker.py:1724 -- Started a local Ray instance.

And the kernel restarts, any thoughts on this?

thanks
tingting

1 reply

SeppeDeWinter Mar 11, 2024
Maintainer

Hi

I'm not sure what is going wrong here, I'm sorry.
This seems to be an issue with ray and your specific set up.

All the best,

Seppe

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TimeoutError from run_cgs_models() in first 20 seconds #127

{{title}}

Replies: 3 comments 10 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

TimeoutError from run_cgs_models() in first 20 seconds #127

galvaniquee Mar 14, 2023

Replies: 3 comments · 10 replies

SeppeDeWinter Mar 14, 2023 Maintainer

SeppeDeWinter Mar 14, 2023 Maintainer

galvaniquee Mar 14, 2023 Author

galvaniquee Mar 14, 2023 Author

SeppeDeWinter Mar 14, 2023 Maintainer

galvaniquee Mar 15, 2023 Author

SeppeDeWinter Mar 15, 2023 Maintainer

galvaniquee Mar 20, 2023 Author

galvaniquee Mar 20, 2023 Author

SeppeDeWinter Mar 27, 2023 Maintainer

TingTingShao Mar 11, 2024

SeppeDeWinter Mar 11, 2024 Maintainer

galvaniquee
Mar 14, 2023

Replies: 3 comments 10 replies

SeppeDeWinter
Mar 14, 2023
Maintainer

SeppeDeWinter Mar 14, 2023
Maintainer

galvaniquee Mar 14, 2023
Author

galvaniquee Mar 14, 2023
Author

SeppeDeWinter Mar 14, 2023
Maintainer

galvaniquee Mar 15, 2023
Author

SeppeDeWinter
Mar 15, 2023
Maintainer

galvaniquee Mar 20, 2023
Author

galvaniquee Mar 20, 2023
Author

SeppeDeWinter Mar 27, 2023
Maintainer

TingTingShao
Mar 11, 2024

SeppeDeWinter Mar 11, 2024
Maintainer