ssh timeout Remote command exited without exit status or exit signal.

[spencer-cdw]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

When filing a bug, please include the following headings if possible. Any
example text in this template can be deleted.

Overview of the Issue

When building ubuntu 20.04 images on GCP, packer always errors with the following message

2023/02/10 20:07:25 packer-plugin-googlecompute_v1.0.16_x5.0_linux_amd64 plugin: 2023/02/10 20:07:25 [DEBUG] starting remote command: echo 'ubuntu' | sudo -S sh -c 'PACKER_BUILDER_TYPE='googlecompute' PACKER_BUILD_NAME='ubuntu-focal'  /tmp/script_5166.sh'
2023/02/10 20:07:25 packer-plugin-googlecompute_v1.0.16_x5.0_linux_amd64 plugin: 2023/02/10 20:07:25 [ERROR] Remote command exited without exit status or exit signal.
2023/02/10 20:07:25 packer-plugin-googlecompute_v1.0.16_x5.0_linux_amd64 plugin: 2023/02/10 20:07:25 [INFO] RPC endpoint: Communicator ended with: 2300218
2023/02/10 20:07:25 [INFO] 0 bytes written for 'stdout'
2023/02/10 20:07:25 [INFO] 0 bytes written for 'stderr'
2023/02/10 20:07:25 [INFO] RPC client: Communicator ended with: 2300218
2023/02/10 20:07:25 [INFO] RPC endpoint: Communicator ended with: 2300218
2023/02/10 20:07:25 packer-provisioner-shell plugin: [INFO] 0 bytes written for 'stdout'
2023/02/10 20:07:25 packer-provisioner-shell plugin: [INFO] 0 bytes written for 'stderr'
2023/02/10 20:07:25 packer-provisioner-shell plugin: [INFO] RPC client: Communicator ended with: 2300218

Note that this does not happen with ubuntu 18.04 or ubuntu 22.04 images, nor does it happen with 20.04 images on AWS or Azure.

Reproduction Steps

The error messages have been brought up before in the following issues:

• SSH timeout error while running packer #4363

Those issues were closed due to being unable to reproduce. The following packer code reproduces the error 100% of the time.

packer {
  required_plugins {
    googlecompute = {
      version = "v1.0.16"
      source  = "github.com/hashicorp/googlecompute"
    }
  }
}

variable "project_id" {
  type    = string
  default = "golden-image-management"
}

source "googlecompute" "ubuntu-focal" {
  project_id             = var.project_id
  source_image           = "ubuntu-2004-focal-v20230125"
  ssh_username           = "packer"
  ssh_read_write_timeout = "5m"
  zone                   = "us-central1-a"
  image_name             = "foobar-ubuntu-20-04-cis1-20230101-01"
}

build {
  name = "ubuntu-focal"
  sources = [
    "source.googlecompute.ubuntu-focal"
  ]

  provisioner "shell" {
    inline          = ["ip a"]
    execute_command = "echo 'ubuntu' | sudo -S sh -c '{{ .Vars }} {{ .Path }}'"
  }

  provisioner "shell" {
    inline            = ["reboot"]
    execute_command   = "echo 'ubuntu' | sudo -S sh -c '{{ .Vars }} {{ .Path }}'"
    expect_disconnect = true
    pause_after       = "120s"
  }

  provisioner "shell" {
    inline          = ["ip a"]
    execute_command = "echo 'ubuntu' | sudo -S sh -c '{{ .Vars }} {{ .Path }}'"
  }
}

Packer version

1.8.5

Log Fragments and crash.log files

https://gist.github.com/spencer-cdw/af6d980d91b2d156f96d1a9c9b1485ac

[spencer-cdw]

Workaround:

force kill sshd before rebooting.

#354 (comment)

  provisioner "shell" {
    # https://github.com/hashicorp/packer/issues/354#issuecomment-23615630
    inline            = ["ps aux | grep sshd | grep -v grep | awk '{print $2}' | xargs kill && reboot"]
    execute_command   = "echo 'ubuntu' | sudo -S sh -c '{{ .Vars }} {{ .Path }}'"
    expect_disconnect = true
    pause_after       = "120s"
  }

[BadJukeBox]

for what it's worth I have the same issue and the workaround does not work for me. Happens while attempting to reboot on AL2.

[cthulhuplus]

I am experiencing this issue on AWS as well. The provisioning process just randomly hangs and after ~20 minutes I get the same error:

2023/04/11 15:06:15 packer-builder-amazon-ebs plugin: [ERROR] Remote command exited without exit status or exit signal.
2023/04/11 15:06:15 packer-builder-amazon-ebs plugin: [INFO] RPC endpoint: Communicator ended with: 2300218
2023/04/11 15:06:15 [INFO] 15064 bytes written for 'stdout'
2023/04/11 15:06:15 [INFO] 48650 bytes written for 'stderr'
2023/04/11 15:06:15 [INFO] RPC client: Communicator ended with: 2300218
2023/04/11 15:06:15 [INFO] RPC endpoint: Communicator ended with: 2300218
2023/04/11 15:06:15 packer-provisioner-shell plugin: [INFO] 15064 bytes written for 'stdout'
2023/04/11 15:06:15 packer-provisioner-shell plugin: [INFO] 48650 bytes written for 'stderr'
2023/04/11 15:06:15 packer-provisioner-shell plugin: [INFO] RPC client: Communicator ended with: 2300218
2023/04/11 15:06:15 [INFO] (telemetry) ending shell

[treyperrone]

Just started building al2023 and seeing this same issue. Anyone ever solve for it?

Using packer 1.11.1

[edlitmus]

FYI: I ran into this issue while running some intensive stages that took a long time to complete. Breaking the stages up into shorter running steps overcame this problem.
It's just a workaround and I don't know the root cause of this bug but this worked for me.

[treyperrone]

I ended up using retry on the shell provisioner and letting it run over again. It doesn't always occur, its perhaps a 50% chance it disconnects. Theres a few more mods i need to make to get it more "idempotent" but most of the things can re-run and just eats up more time...