Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[v8.0] fix (DISET): Resolve infinite loop in DISET for DISET.BaseTransport sendData #7848

Merged
merged 1 commit into from
Oct 23, 2024

Conversation

ryuwd
Copy link

@ryuwd ryuwd commented Oct 23, 2024

We noticed that the socket in DISET BaseTransport can get into a state where self.oSocket.send(buf) will always return -1 (as shown in below trace) - this causes packSentBytes to decrement, and if the write operation always returns a negative value we end up in an infinite loop as packSentBytes < bytesToSend can never be false.

cc @chrisburr

Thread 458188 (active): "ThreadPoolExecutor-0_14"
    _write_bio (M2Crypto/SSL/Connection.py:352)
        Arguments:
            self: <Connection at 0x7f09746fe050>
            data: <bytes at 0x7f092c0440f0>
    write (M2Crypto/SSL/Connection.py:373)
        Arguments:
            self: <Connection at 0x7f09746fe050>
            data: <bytes at 0x7f092c0440f0>
    _write (DIRAC/Core/DISET/private/Transports/M2SSLTransport.py:499)
        Arguments:
            self: <SSLTransport at 0x7f09746fef50>
            buf: <bytes at 0x7f092c0440f0>
    sendData (DIRAC/Core/DISET/private/Transports/BaseTransport.py:173)
        Arguments:
            self: <SSLTransport at 0x7f09746fef50>
            uData: {"OK": False, "Errno": 0, "Message": "Invalid action proposal", "CallStack": ["  File \"/opt/dirac/versions/v11.0.49-1729166531/Linux-x86_64/lib/python3.11/threading.py\", line 1002, in _bootstrap\n    self._bootstrap_inner()\n", "  File \"/opt/dirac/versions/v11.0.49-1729166531/Linux-x86_64/lib/python3.11/threading.py\", line 1045, in _bootstrap_inner\n    self.run()\n", "  File \"/opt/dirac/versions/v11.0.49-1729166531/Linux-x86_64/lib/python3.11/threading.py\", line 982, in run\n    self._target(*self._args, **self._kwargs)\n", "  File \"/opt/dirac/versions/v11.0.49-1729166531/Linux-x86_64/lib/python3.11/concurrent/futures/thread.py\", line 83, in _worker\n    work_item.run()\n", "  File \"/opt/dirac/versions/v11.0.49-1729166531/Linux-x86_64/lib/python3.11/concurrent/futures/thread.py\", line 58, in run\n    result = self.fn(*self.args, **self.kwargs)\n", "  File \"/opt/dirac/versions/v11.0.49-1729166531/Linux-x86_64/lib/python3.11/site-packages/DIRAC/Core/DISET/private/Service.py\", line 337, in _processInThread\n    result = self._receiveAndCheckProposal(trid)\n", "  File \"/opt/dirac/versions/v11.0.49-1729166531/Linux-x86_64/lib/python3.11/site-packages/DIRAC/Core/DISET/private/Service.py\", line 397, in _receiveAndCheckProposal\n    return S_ERROR(\"Invalid action proposal\")\n"]}
            prefix: <bytes at 0x55a820b753a8>
        Locals:
            sCodedData: <bytes at 0x7f092c019390>
            dataToSend: <bytes at 0x7f092c0440f0>
            index: 0
            bytesToSend: 1283
            packSentBytes: -180929945790
            result: {"OK": True, "Value": -1}
            sentBytes: -1
    sendAndClose (DIRAC/Core/DISET/private/TransportPool.py:125)
        Arguments:
            self: <TransportPool at 0x7f0986a0f390>
            trid: "snip"
            msg: {"OK": False, "Errno": 0, "Message": "Invalid action proposal", "CallStack": ["  File \"/opt/dirac/versions/v11.0.49-1729166531/Linux-x86_64/lib/python3.11/threading.py\", line 1002, in _bootstrap\n    self._bootstrap_inner()\n", "  File \"/opt/dirac/versions/v11.0.49-1729166531/Linux-x86_64/lib/python3.11/threading.py\", line 1045, in _bootstrap_inner\n    self.run()\n", "  File \"/opt/dirac/versions/v11.0.49-1729166531/Linux-x86_64/lib/python3.11/threading.py\", line 982, in run\n    self._target(*self._args, **self._kwargs)\n", "  File \"/opt/dirac/versions/v11.0.49-1729166531/Linux-x86_64/lib/python3.11/concurrent/futures/thread.py\", line 83, in _worker\n    work_item.run()\n", "  File \"/opt/dirac/versions/v11.0.49-1729166531/Linux-x86_64/lib/python3.11/concurrent/futures/thread.py\", line 58, in run\n    result = self.fn(*self.args, **self.kwargs)\n", "  File \"/opt/dirac/versions/v11.0.49-1729166531/Linux-x86_64/lib/python3.11/site-packages/DIRAC/Core/DISET/private/Service.py\", line 337, in _processInThread\n    result = self._receiveAndCheckProposal(trid)\n", "  File \"/opt/dirac/versions/v11.0.49-1729166531/Linux-x86_64/lib/python3.11/site-packages/DIRAC/Core/DISET/private/Service.py\", line 397, in _receiveAndCheckProposal\n    return S_ERROR(\"Invalid action proposal\")\n"]}
    _processInThread (DIRAC/Core/DISET/private/Service.py:339)
        Arguments:
            self: <Service at 0x7f0986a8d050>
            clientTransport: <SSLTransport at 0x7f09746fef50>
        Locals:
            monReport: (snip)
            result: {"OK": False, "Errno": 0, "Message": "Invalid action proposal", "CallStack": ["  File \"/opt/dirac/versions/v11.0.49-1729166531/Linux-x86_64/lib/python3.11/threading.py\", line 1002, in _bootstrap\n    self._bootstrap_inner()\n", "  File \"/opt/dirac/versions/v11.0.49-1729166531/Linux-x86_64/lib/python3.11/threading.py\", line 1045, in _bootstrap_inner\n    self.run()\n", "  File \"/opt/dirac/versions/v11.0.49-1729166531/Linux-x86_64/lib/python3.11/threading.py\", line 982, in run\n    self._target(*self._args, **self._kwargs)\n", "  File \"/opt/dirac/versions/v11.0.49-1729166531/Linux-x86_64/lib/python3.11/concurrent/futures/thread.py\", line 83, in _worker\n    work_item.run()\n", "  File \"/opt/dirac/versions/v11.0.49-1729166531/Linux-x86_64/lib/python3.11/concurrent/futures/thread.py\", line 58, in run\n    result = self.fn(*self.args, **self.kwargs)\n", "  File \"/opt/dirac/versions/v11.0.49-1729166531/Linux-x86_64/lib/python3.11/site-packages/DIRAC/Core/DISET/private/Service.py\", line 337, in _processInThread\n    result = self._receiveAndCheckProposal(trid)\n", "  File \"/opt/dirac/versions/v11.0.49-1729166531/Linux-x86_64/lib/python3.11/site-packages/DIRAC/Core/DISET/private/Service.py\", line 397, in _receiveAndCheckProposal\n    return S_ERROR(\"Invalid action proposal\")\n"]}
            trid: "snip"
    run (concurrent/futures/thread.py:58)
        Arguments:
            self: <_WorkItem at 0x7f09746fdf10>
    _worker (concurrent/futures/thread.py:83)
        Arguments:
            executor_reference: <weakref.ReferenceType at 0x7f0984891f30>
            work_queue: <_queue.SimpleQueue at 0x7f098650aac0>
            initializer: None
            initargs: ()
        Locals:
            work_item: <_WorkItem at 0x7f09746fdf10>
    run (threading.py:982)
        Arguments:
            self: <Thread at 0x7f09865d3050>
    _bootstrap_inner (threading.py:1045)
        Arguments:
            self: <Thread at 0x7f09865d3050>
    _bootstrap (threading.py:1002)
        Arguments:
            self: <Thread at 0x7f09865d3050>

BEGINRELEASENOTES

*Core
FIX: Prevent -1 returned from oSocket.write being interpreted as number of bytes to avoid an infinite loop in DISET BaseTransport.sendData.

ENDRELEASENOTES

@ryuwd ryuwd changed the title Resolve infinite loop in DISET for DISET.BaseTransport sendData fix (DISET): Resolve infinite loop in DISET for DISET.BaseTransport sendData Oct 23, 2024
chrisburr
chrisburr previously approved these changes Oct 23, 2024
@ryuwd ryuwd changed the title fix (DISET): Resolve infinite loop in DISET for DISET.BaseTransport sendData [v8.0] fix (DISET): Resolve infinite loop in DISET for DISET.BaseTransport sendData Oct 23, 2024
@ryuwd ryuwd changed the base branch from integration to rel-v8r0 October 23, 2024 09:43
@fstagni fstagni merged commit 418c605 into DIRACGrid:rel-v8r0 Oct 23, 2024
2 checks passed
@DIRACGridBot DIRACGridBot added the sweep:ignore Prevent sweeping from being ran for this PR label Oct 23, 2024
@ryuwd
Copy link
Author

ryuwd commented Oct 23, 2024

@chrisburr is this missing a sweep to integration branch?

@chaen
Copy link
Contributor

chaen commented Oct 28, 2024

An old friend :-) #7462

@fstagni fstagni added alsoTargeting:integration Cherry pick this PR to integration after merge and removed sweep:ignore Prevent sweeping from being ran for this PR labels Oct 31, 2024
@DIRACGridBot DIRACGridBot added the sweep:done All sweeping actions have been done for this PR label Oct 31, 2024
DIRACGridBot pushed a commit to DIRACGridBot/DIRAC that referenced this pull request Oct 31, 2024
@DIRACGridBot
Copy link

Sweep summary

Sweep ran in https://github.com/DIRACGrid/DIRAC/actions/runs/11608724944

Successful:

  • integration

fstagni added a commit to DIRACGridBot/DIRAC that referenced this pull request Nov 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
alsoTargeting:integration Cherry pick this PR to integration after merge sweep:done All sweeping actions have been done for this PR
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants