Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SimpleQueue use differs across python versions #1023

Open
3 tasks done
davidp1404 opened this issue Jan 29, 2025 · 0 comments
Open
3 tasks done

SimpleQueue use differs across python versions #1023

davidp1404 opened this issue Jan 29, 2025 · 0 comments
Labels

Comments

@davidp1404
Copy link

Things to check first

  • I have checked that my issue does not already have a solution in the FAQ

  • I have searched the existing issues and didn't find my bug already reported there

  • I have checked that my bug is still present in the latest release

Version

APScheduler==3.10.4

What happened?

Using this simple code:

import asyncio
from multiprocessing import SimpleQueue, set_start_method
from apscheduler.executors.pool import ThreadPoolExecutor, ProcessPoolExecutor
from apscheduler.schedulers.background import BackgroundScheduler
from random import random
import time
import uuid
from confluent_kafka import Consumer, KafkaException


# Function to be run as a daemon
def daemon_task_1(**kwargs):
    print("Daemon Task 1 is running")
    while True:
        queue.put(random())

def daemon_task_2(**kwargs):
    print("Daemon Task 2 is running")
    while True:
        data = queue.get()
        print(data)


def main():
    executors = {
        'default': ProcessPoolExecutor(max_workers=8,)
    }
    job_defaults = {
        'coalesce': False,
        'max_instances': 3,
        'misfire_grace_time': None
    }    
    # Scheduler for the daemon tasks
    scheduler = BackgroundScheduler(
        executors=executors,
        job_defaults=job_defaults,
        daemonic=True,
        daemon=True
    )
    
    scheduler.add_job(daemon_task_1, trigger='date')
    scheduler.add_job(daemon_task_2, trigger='date')

    global queue
    queue = SimpleQueue()
    # Start the scheduler
    scheduler.start()

    try:
        while True:
            time.sleep(1)
    except (KeyboardInterrupt, SystemExit):
        pass
    finally:
        scheduler.shutdown()

if __name__ == '__main__':
    main()

When I run it with python 3.8 it works fine:

$ python -V;python aps-mixed.py | more
Python 3.8.18
Daemon Task 2 is running
0.7147972065089533
0.9088275181513735
0.770493801418164
0.0914193457803868
0.7720603694858935
0.48302319930441084
...

But with 3.11 I get this error, why?

Python 3.11.11
Job "daemon_task_1 (trigger: date[2025-01-29 16:47:01 CET], next run at: 2025-01-29 16:47:01 CET)" raised an exception
Traceback (most recent call last):
  File "/home/davidp/.pyenv/versions/mlserver2/lib/python3.11/site-packages/apscheduler/executors/base.py", line 131, in run_job
    retval = job.func(*job.args, **job.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/davidp/gitlab/geoapps-ta-study/kafka/aps-mixed.py", line 15, in daemon_task_1
    queue.put(random())
    ^^^^^
NameError: name 'queue' is not defined
Job "daemon_task_2 (trigger: date[2025-01-29 16:47:01 CET], next run at: 2025-01-29 16:47:01 CET)" raised an exception
Traceback (most recent call last):
  File "/home/davidp/.pyenv/versions/mlserver2/lib/python3.11/site-packages/apscheduler/executors/base.py", line 131, in run_job
    retval = job.func(*job.args, **job.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/davidp/gitlab/geoapps-ta-study/kafka/aps-mixed.py", line 20, in daemon_task_2
    data = queue.get()
           ^^^^^
NameError: name 'queue' is not defined

How can we reproduce the bug?

Already described in the issue

@davidp1404 davidp1404 added the bug label Jan 29, 2025
@davidp1404 davidp1404 changed the title SimpleQueue sharing acroos multiprocessing differ across python versions SimpleQueue use differs across python versions Jan 29, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant