Replies: 2 comments 11 replies
-
|
Definitely sounds like a bug in Moulti. I may have overlooked something re: Ansible parallelism. It may also be an issue with the way Moulti reads data from Something that does not hurt and that you can try is setting Overall, that looks like a reproducible issue. Ideally, you should be able to turn your playbook into a minimal reproducible example that you could share without revealing anything relevant. I would also need your Beyond that, what to look at to diagnose what happens:
|
Beta Was this translation helpful? Give feedback.
-
|
Yeah, SIP is a PITA, I understand. |
Beta Was this translation helpful? Give feedback.
-
|
The lockup is always reproducible but very hard to isolate into a minimal reproducible example. When I create a small example, nothing goes wrong. So there is some sort of strange interaction that's dependent on previous steps (which number in the dozens and would take much more time to properly diagnose). But here's what I have so far, which may be helpful. This is what I'm invoking: MOULTI_ANSIBLE=force MOULTI_INSTANCE=ansible-$$ moulti run -- ansible-playbook site.book.ymland for one of my hosts, my playbook locks up on this rule: Here's the interesting part: I looked at my [
{
"key_url": "https://packagecloud.io/ookla/speedtest-cli/gpgkey",
"keyring": "/etc/apt/keyrings/ookla_speedtest-cli-archive-keyring.gpg",
"package": "speedtest",
"repo": "deb [signed-by=/etc/apt/keyrings/ookla_speedtest-cli-archive-keyring.gpg] https://packagecloud.io/ookla/speedtest-cli/ubuntu/ jammy main",
"sources_list_name": "ookla_speedtest-cli"
},
{
"key_url": "https://repo.charm.sh/apt/gpg.key",
"keyring": "/etc/apt/keyrings/charm.gpg",
"package": "glow",
"repo": "deb [signed-by=/etc/apt/keyrings/charm.gpg] https://repo.charm.sh/apt/ * *",
"sources_list_name": "charm"
},
{
"key_url": "https://cli.github.com/packages/githubcli-archive-keyring.gpg",
"keyring": "/usr/share/keyrings/githubcli-archive-keyring.gpg",
"package": "gh",
"repo": "deb [signed-by=/usr/share/keyrings/githubcli-archive-keyring.gpg\n arch=arm64]\nhttps://cli.github.com/packages stable main",
"sources_list_name": "github-cli"
},
[…]
]And the playbook locks up as long as there are 3 or more entries—i.e. the loop goes for 3 iterations or more. 1 or 2 works just fine. And it doesn't matter what the content of each loop iteration/item is: even if I have the same item 3 times, the lockup happens if the loop has to do 3+ iterations. Again if I try to isolate this in a minimal reproducible example, I can't trigger the lockup. Btw, toggling So there's some complex interaction going on that is hard to diagnose |
Beta Was this translation helpful? Give feedback.
-
|
Thank you -- this is very interesting. I will try to tinker with it. |
Beta Was this translation helpful? Give feedback.
-
|
Additionally, which exact version of these do you use?
|
Beta Was this translation helpful? Give feedback.
-
|
I hadn't enabled I'm running:
❯ ansible-playbook --version
ansible-playbook [core 2.17.2]
config file = /Users/huyz/.ansible.cfg
configured module search path = ['/Users/huyz/.ansible/plugins/modules', '/opt/local/share/ansible/plugins/modules']
ansible python module location = /opt/local/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/ansible
ansible collection location = /Users/huyz/.ansible/collections:/opt/local/share/ansible/collections
executable location = /Users/huyz/bin/ansible-playbook
python version = 3.11.9 (main, Apr 3 2024, 23:19:16) [Clang 15.0.0 (clang-1500.1.0.2.5)] (/opt/local/Library/Frameworks/Python.framework/Versions/3.11/bin/python3.11)
jinja version = 3.1.4
libyaml = True └─┬= 69514 huy /opt/local/Library/Frameworks/Python.framework/Versions/3.11/Resources/Python.app/Contents/MacOS/Python /opt/local/Library/Frameworks/Python.framework/Versions/3.11/bin/moulti run -- ansible-playbook site.book.yml -vvv
└─┬─ 69523 huy /opt/local/Library/Frameworks/Python.framework/Versions/3.11/Resources/Python.app/Contents/MacOS/Python /Users/huyz/bin/ansible-playbook site.book.yml -vvv
├─── 69525 huy /opt/local/Library/Frameworks/Python.framework/Versions/3.11/Resources/Python.app/Contents/MacOS/Python -c from multiprocessing.resource_tracker import main;main(10)
├─── 73306 huy /opt/local/Library/Frameworks/Python.framework/Versions/3.11/Resources/Python.app/Contents/MacOS/Python /opt/local/Library/Frameworks/Python.framework/Versions/3.11/bin/moulti pass ansible_69523_47
└─── 73308 huy /opt/local/Library/Frameworks/Python.framework/Versions/3.11/Resources/Python.app/Contents/MacOS/Python /Users/huyz/bin/ansible-playbook site.book.yml -vvv |
Beta Was this translation helpful? Give feedback.
-
|
Not that simple to reproduce: I gave it a first try using 10 VMs and dummy tasks (like copying a small text file or sleeping a random number of seconds): everything went smoothly.
|
Beta Was this translation helpful? Give feedback.
-
|
Wow, thanks for trying to diagnose it even with the little information I gave you. I will try to give you more info. Just a bit busy these days. Please bear with me :) |
Beta Was this translation helpful? Give feedback.
-
|
Could there be a link between this discussion and issue #16? For instance, Ansible would ask a password through the tty, but the prompt would be erased by a screen refresh, leaving you with the feeling that the whole thing got stuck. A possible side-effect is that you would need to press |
Beta Was this translation helpful? Give feedback.
-
|
Ok let me take a look today and see if I can diagnose and gather more details. Thanks for your patience! |
Beta Was this translation helpful? Give feedback.
-
|
It doesn't look like a password prompt is related based on my tests |
Beta Was this translation helpful? Give feedback.
-
|
Fair enough. If you find some time for this matter, try to turn your playbook into a MRE. |
Beta Was this translation helpful? Give feedback.
-
I used to have one host (out of about 8) that got stuck during one of the steps (always the same one). So I would exclude moulti for that host.
After some recent moulti updates, it now seems to work.
However, now I have two other hosts that used to work but now get stuck (always the same step for each of the hosts, although the steps are different between the hosts).
On the remote host, when I try to see what's going on by running a sort of "ansible watch":
… I see that nothing's going on at the remote host that I can detect.
Now, if I quit moulti with
q t, then that seems to kickstart things up again as I see orphaned output spewing out in my terminal (and my "ansible watch" starts showing new steps again):If I run the Ansible playbooks without moulti, I have no such freezing problem.
I'm trying to diagnose and collect more info about what could be going wrong. For each susceptible Ansible playbook, looking at the Ansible step that looks responsible for locking things up, I can't discern anything special about the step. They're very ordinary steps, e.g. am
ansible.builtin.copyorAny tips on how else I could diagnose this? Maybe run moulti in debugging mode?
Beta Was this translation helpful? Give feedback.
All reactions