execute Ansible task with Teleport (K8s deployment) using Tunnel for connection

[pankajpandey9]

Hi All,
I have Teleport Cluster deployed on Kubernetes and I need to execute Ansible tasks to other VMs using Teleport tunnel.
Since I am using tunnel for Node to Proxy (port 443). Which has no 3022 port open on the host (so cant specify the port in ssh config / Ansible config) files
And if I use proxy port as 3025 for node addition then the node is adds sucessfully. However, cant connect via UI or tsh and throws error "dialing directly: dial tcp 10.x.x.0:3022: connect: connection refused". where the IP address is from Kubernetes Cluster and does not belong to the node / proxy / auth
Please advise... Thanks

[webvictim]

You should be able to do this with a custom ssh.cfg file and a custom ansible.cfg.

In the configs below:

• Replace iot-node with your IoT node's hostname (this must match the hostname showing in the Teleport web UI that you use with tsh ssh iot-node so Teleport knows how to tunnel to it)
• Replace proxyuser with a UNIX login that is allowed to log into your proxy (i.e. your Teleport role must grant permission for that login)
• Replace nodeuser with a UNIX login that is allowed to log into your node (i.e. your Teleport role must grant permission for that login)
• Replace teleport.example.com with the public address of your Teleport proxy
• Before running Ansible, run tsh login --proxy=teleport.example.com and make sure that you have successfully authenticated
  • Also check that the Teleport certificate is loaded into your ssh-agent so Ansible can use it:

$ ssh-add -l
2048 SHA256:oHxI4xZcGS6SsHLl7G9UMnSAZR9Cr4Qc8T7axe8FjhE teleport:teleportusername (RSA-CERT)
2048 SHA256:oHxI4xZcGS6SsHLl7G9UMnSAZR9Cr4Qc8T7axe8FjhE teleport:teleportusername (RSA)

• Get the public key of your Teleport host CA by using tctl auth export --type=host on your Teleport auth server and write this to ssh_known_hosts:

$ cat ssh_known_hosts
@cert-authority teleport.example.com,*.teleport.example.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDX2iIvsuLDdAqd...<replace this line with the output from your command>

ssh.cfg:

Host iot-node
  ProxyCommand ssh -F ./ssh.cfg -p 3023 [email protected] -s proxy:%h:%p
  Port 3022
  User nodeuser
  UserKnownHostsFile ./ssh_known_hosts

Host teleport.example.com
  Port 3023
  User proxyuser
  ControlMaster auto
  ControlPath ~/.ssh/ansible-%r@%h:%p
  ControlPersist 5m
  UserKnownHostsFile ./ssh_known_hosts

ansible.cfg:

[ssh_connection]
ssh_args = -F ./ssh.cfg -o ControlMaster=auto -o ControlPersist=5m -o UserKnownHostsFile=./ssh_known_hosts
control_path = ~/.ssh/ansible-%%r@%%h:%%p
scp_if_ssh = True

At this point you should be able to run ansible -v all -i inventory_file -m ping and get a response:

iot-node | SUCCESS => {
    "changed": false,
    "ping": "pong"
}

[pankajpandey9]

Hi @webvictim,

I followed the instructions above however, evertime I am getting the following error

dev-teleport-a is the VM
dev-teleport is the Teleport Cluster host
It seems infraadmini user is not allowed to authenticate on the Teleport server on 3023

infraadmin@dev-teleport: Permission deniedPermission denied (publickey).
kex_exchange_identification: Connection closed by remote host

infraadmin user is available on both VM and the K8s Teleport (Local user part to admin group)
Also tried with Port 3022 and 22 port for the VM
Added all the host entries with hostname, IP and DNS for the VM dev-teleport-a in known_hosts and .ssh/config file

~/.ssh/config

Host dev-teleport-a
  ProxyCommand ssh -F ~/.ssh/config -p 3023 infraadmin@dev-teleport -s proxy:%h:%p
  Port 3022 
  User infraadmin
  UserKnownHostsFile ~/.ssh/known_hosts

Host dev-teleport
  Port 3023
  User infraadmin
  ControlMaster auto
  ControlPath ~/.ssh/ansible-%r@%h:%p
  ControlPersist 5m
  UserKnownHostsFile ~/.ssh/known_hosts

~/ansible.cfg

[ssh_connection]
ssh_args = -F ./ssh.cfg -o ControlMaster=auto -o ControlPersist=5m -o UserKnownHostsFile=~/.ssh/known_hosts
control_path = ~/.ssh/ansible-%%r@%%h:%%p
scp_if_ssh = True

~/.ssh/known_hosts
@cert-authority dev-teleport-a ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCy8NBSdLuI49wwIY1Ty80836hhsYmlyPmMpiYPDjgAS0z0KcIjmO7qb98622sKqj1g+ logins=-teleport-nologin-eab51=host

[webvictim]

@pankajpandey9 Can you show the output of tsh status after logging into your Teleport cluster? Does it say infraadmin next to Logins:? If not, you will need to make sure that your Teleport user has a Teleport role which grants the infraadmin login.

I also noticed that your @cert-authority line only contains dev-teleport-a - it should start with @cert-authority dev-teleport,dev-teleport-a ssh-rsa... to make sure that host CA fingerprint is used to match against both those hosts. Without this, you will get a certificate invalid: name is not a listed principal error when logging in.

[pankajpandey9]

@webvictim ,

I can login using infraadmin (local user and not oidc)

[pankaj.pandey@dev-bastion ~]$ tsh login --proxy=dev-teleport:443 --user=infraadmin --auth local
Enter password for Teleport user infraadmin:
Enter your OTP token:
612285
> Profile URL:        https://dev-teleport:3080
  Logged in as:       infraadmin
  Cluster:            dev-teleport
  Roles:              admins*
  Logins:             infraadmin
  Kubernetes:         disabled
  Valid until:        2021-04-21 19:41:15 -0500 CDT [valid for 12h0m0s]
  Extensions:         permit-port-forwarding, permit-pty

I have added the @cert entries individually for all hosts (Server and Node)

@cert-authority dev-teleport-a ssh-rsa
@cert-authority <IP Address> ssh-rsa
@cert-authority dev-teleport ssh-rsa

Here is the redacted output while doing SSH

[pankaj.pandey@dev-bastion ~]$ ssh infraadmin@dev-teleport-a -v
FIPS mode initialized
OpenSSH_8.0p1, OpenSSL 1.1.1c FIPS  28 May 2019
debug1: /home/pankaj.pandey/.ssh/config line 32: Applying options for dev-teleport-a
debug1: configuration requests final Match pass
debug1: re-parsing configuration
debug1: Reading configuration data /home/pankaj.pandey/.ssh/config
debug1: /home/pankaj.pandey/.ssh/config line 32: Applying options for dev-teleport-a
—
debug1: Executing proxy command: exec ssh -F ~/.ssh/config -p 3023 infraadmin@dev-teleport -s proxy:dev-teleport-a:3022
-
debug1: Local version string SSH-2.0-OpenSSH_8.0
FIPS mode initialized
debug1: Remote protocol version 2.0, remote software version Teleport
debug1: no match: Teleport
debug1: Authenticating to dev-teleport-a:3022 as 'infraadmin'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: ecdh-sha2-nistp256
debug1: kex: host key algorithm: [email protected]
debug1: kex: server->client cipher: aes256-ctr MAC: [email protected] compression: none
debug1: kex: client->server cipher: aes256-ctr MAC: [email protected] compression: none
debug1: kex: ecdh-sha2-nistp256 need=32 dh_need=32
debug1: kex: ecdh-sha2-nistp256 need=32 dh_need=32
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host certificate: [email protected] SHA256:<some value>, serial 0 ID "" CA ssh-rsa SHA256:<some value> valid after 2021-04-06T06:44:33
debug1: checking without port identifier
debug1: Host 'dev-teleport1-a' is known and matches the RSA-CERT host certificate.
debug1: Found CA key in /home/pankaj.pandey/.ssh/known_hosts:27
debug1: found matching key w/out port
debug1: rekey out after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 4294967296 blocks
debug1: Will attempt key: /home/pankaj.pandey/teleport_infraadmin/infraadmin RSA-CERT SHA256:<some value> explicit
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /home/pankaj.pandey/teleport_infraadmin/infraadmin RSA-CERT SHA256:<some value> explicit
debug1: send_pubkey_test: no mutual signature algorithm
debug1: No more authentication methods to try.
infraadmin@dev-teleport-a: Permission denied (publickey).

[webvictim]

debug1: send_pubkey_test: no mutual signature algorithm

This looks like there's no mutual signature algorithm which is agreeable between the two servers. I suspect this is due to support for ssh-rsa based certificates being removed in recent versions of OpenSSH.

There are two possible fixes:

1. Here's a link with some more details and advice on how you can change the OpenSSH server's configuration to trust ssh-rsa signatures again: https://confluence.atlassian.com/bitbucketserverkb/ssh-rsa-key-rejected-with-message-no-mutual-signature-algorithm-1026057701.html

2. Alternatively, you can change the CA signature algorithm that Teleport uses and rotate the CA. Change the Teleport auth server's config file:

teleport:
  ca_signature_algo: "rsa-sha2-512"

After this, restart Teleport and then rotate the certificate authority with tctl auth rotate --grace-period=1m (note that this will cause existing certificates to become invalid after the grace period expires)

Once the rotation is complete, get the new output of tctl auth export --type=user, write this to ~/.ssh/authorized_keys and try again. The server should accept the newer certificate authority. You'll also need to re-run tctl auth export --type=host and update your known_hosts file with the new fingerprint.

[pankajpandey9]

@webvictim , Thanks for the reply, I will try the suggestions and will update