Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Resource deadlock on ACK container deployment. #223

Open
mtaruno opened this issue Jan 14, 2025 · 1 comment
Open

[BUG] Resource deadlock on ACK container deployment. #223

mtaruno opened this issue Jan 14, 2025 · 1 comment
Labels
bug Something isn't working

Comments

@mtaruno
Copy link

mtaruno commented Jan 14, 2025

Describe the bug
I'm having problems on the ACK side. I've modified the deploy.sh to become more verbose, doing:

az acr build --debug --registry $containerRegistry \
        --file "$projectRoot/docker/Dockerfile-backend" \
        --image $GRAPHRAG_IMAGE \
        "$projectRoot" || {
            errorBanner
            echo "Failed to build and push image to ACR"
            exit 1
        }

However, I still get this issue (output of the verbose command):

...
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'backend/src/__init__.py'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'backend/src/utils'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'backend/src/utils/query.py'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'backend/src/utils/graph.py'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'backend/src/utils/__init__.py'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'backend/src/utils/workflows.py'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'backend/src/typing'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'backend/src/typing/__init__.py'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'backend/src/typing/pipeline.py'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'backend/src/api'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'backend/src/api/pipeline-settings.yaml'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'backend/src/api/query_streaming.py'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'backend/src/api/query.py'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'backend/src/api/index.py'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'backend/src/api/index_configuration.py'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'backend/src/api/graph.py'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'backend/src/api/__init__.py'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'backend/src/api/common.py'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'backend/src/api/source.py'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'backend/src/api/data.py'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'backend/src/api/azure_clients.py'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'backend/src/main.py'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'docs'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'docs/DEPLOYMENT-GUIDE.md'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'docs/DEVELOPMENT-GUIDE.md'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'docs/assets'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'docs/assets/graphrag-architecture-diagram.vsdx'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'docs/assets/graphrag-architecture-diagram.png'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for '.devcontainer'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for '.devcontainer/Dockerfile'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for '.devcontainer/entrypoint.sh'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for '.devcontainer/devcontainer.json'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for '.editorconfig'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'hestia'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'hestia/bin'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'hestia/bin/jupyter-run'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'hestia/bin/Activate.ps1'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'hestia/bin/python3'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'hestia/bin/pip3.12'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'hestia/bin/python'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'hestia/bin/debugpy'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'hestia/bin/pip3'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'hestia/bin/ipython'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'hestia/bin/activate.fish'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'hestia/bin/ipython3'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'hestia/bin/pip'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'hestia/bin/pip3.12 2'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'hestia/bin/tqdm'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'hestia/bin/jupyter-troubleshoot'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'hestia/bin/pygmentize'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'hestia/bin/jupyter-migrate'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'hestia/bin/activate'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'hestia/bin/normalizer'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'hestia/bin/python3.12'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'hestia/bin/jupyter-kernelspec'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'hestia/bin/jupyter-kernel'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'hestia/bin/jupyter'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'hestia/bin/activate.csh'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'hestia/include'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'hestia/include/python3.12'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'hestia/pyvenv.cfg'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'hestia/lib'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'hestia/lib/python3.12'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'hestia/lib/python3.12/site-packages'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'hestia/lib/python3.12/site-packages/appnope'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'hestia/lib/python3.12/site-packages/appnope/_nope.py'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'hestia/lib/python3.12/site-packages/appnope/__init__.py'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'hestia/lib/python3.12/site-packages/appnope/__pycache__'. parent ignore 'False'
cli.azure.cli.command_modules.acr._archive_utils: .dockerignore: no rule for 'hestia/lib/python3.12/site-packages/appnope/__pycache__/_nope.cpython-312.pyc'. parent ignore 'False'
cli.azure.cli.command_modules.acr.build: Deleting the archived source code from '/tmp/build_archive_f9be77357d88462bab3ef22dd1f19c86.tar.gz'...
cli.azure.cli.core.azclierror: Traceback (most recent call last):
  File "/opt/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/build.py", line 72, in acr_build
    source_location = upload_source_code(
                      ^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/_archive_utils.py", line 28, in upload_source_code
    _pack_source_code(source_location,
  File "/opt/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/_archive_utils.py", line 99, in _pack_source_code
    _archive_file_recursively(tar,
  File "/opt/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/_archive_utils.py", line 208, in _archive_file_recursively
    _archive_file_recursively(tar, os.path.join(name, f), os.path.join(arcname, f),
  File "/opt/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/_archive_utils.py", line 208, in _archive_file_recursively
    _archive_file_recursively(tar, os.path.join(name, f), os.path.join(arcname, f),
  File "/opt/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/_archive_utils.py", line 208, in _archive_file_recursively
    _archive_file_recursively(tar, os.path.join(name, f), os.path.join(arcname, f),
  [Previous line repeated 4 more times]
  File "/opt/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/_archive_utils.py", line 201, in _archive_file_recursively
    tar.addfile(tarinfo, f)
  File "/opt/az/lib/python3.12/tarfile.py", line 2241, in addfile
    copyfileobj(fileobj, self.fileobj, tarinfo.size, bufsize=bufsize)
  File "/opt/az/lib/python3.12/tarfile.py", line 257, in copyfileobj
    buf = src.read(remainder)
          ^^^^^^^^^^^^^^^^^^^
OSError: [Errno 35] Resource deadlock avoided

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/az/lib/python3.12/site-packages/knack/cli.py", line 233, in invoke
    cmd_result = self.invocation.execute(args)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py", line 666, in execute
    raise ex
  File "/opt/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py", line 733, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py", line 703, in _run_job
    result = cmd_copy(params)
             ^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py", line 336, in __call__
    return self.handler(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.12/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
    return op(**command_args)
           ^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/build.py", line 80, in acr_build
    raise CLIError(err)
knack.util.CLIError: [Errno 35] Resource deadlock avoided

cli.azure.cli.core.azclierror: [Errno 35] Resource deadlock avoided
az_command_data_logger: [Errno 35] Resource deadlock avoided
cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0xffff91c6af20>]
az_command_data_logger: exit code: 1
cli.__main__: Command ran in 24.368 seconds (init: 0.064, invoke: 24.304)
telemetry.main: Begin splitting cli events and extra events, total events: 1
telemetry.client: Accumulated 0 events. Flush the clients.
telemetry.main: Finish splitting cli events and extra events, cli events: 1
telemetry.save: Save telemetry record of length 3903 in cache file under /home/vscode/.azure/telemetry/20250114084440237
telemetry.main: Begin creating telemetry upload process.
telemetry.process: Creating upload process: "/opt/az/bin/python3 /opt/az/lib/python3.12/site-packages/azure/cli/telemetry/__init__.py /home/vscode/.azure /home/vscode/.azure/telemetry/20250114084440237"
telemetry.process: Return from creating process 55481
telemetry.main: Finish creating telemetry upload process.
+ errorBanner
+ cat
 ________________________________
/  Uh oh, an error has occurred. \
\  Please see message below.     /
 ‾‾‾‾‾‾‾‾‾‾/‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾
          /
      __ /
     /  \
    ~    ~
   / \  /_\
   \o/  \o/
    |    |
    ||   |/
    ||   ||
    ||   ||
    | \_/ |
    \     /
     \___/
+ printf '\n'

+ echo 'Failed to build and push image to ACR'
Failed to build and push image to ACR
+ exit 1

Expected behavior
I want the deploy.sh to finish running successfully until I get the success banner.

Desktop (please complete the following information):

  • MacOS Sonoma 14.4.1

Additional context
I made a venv on the actual repo, but I have already added it aggressively to the .dockerignore with these:
.venv/
venv/
env/
./hestia
.hestia/
hestia/
**/hestia/
/hestia/
**/.venv/
/.venv/

Anybody have any idea what's going on? Thank you in advance.

@mtaruno mtaruno added the bug Something isn't working label Jan 14, 2025
@itismejy
Copy link

Having the same issue, hopefully it gets resolved

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants