sslh and stunnel configuration not working

[baptx]

Hello, I would like to use SSH on port 443 and encapsulate the traffic with SSL in case I will use an Internet connection that only allows port 443 and SSL traffic.

I started sslh and stunnel like that (based on the example https://github.com/yrutschle/sslh/blob/master/doc/config.md#using-proxytunnel-with-sslh):
sudo stunnel /etc/stunnel/stunnel.conf -f -d vps:443 -l /usr/sbin/sslh -- sslh -i --ssh localhost:22
The domain name "vps" is defined in the file /etc/hosts of the server and points to the public IPv4 and IPv6 address of my VPS.
Compared to the example, did not need to use --http localhost:80 and I had to add the configuration file /etc/stunnel/stunnel.conf with stunnel4, otherwise there is an error. Here is what the configuration file contains:

connect = logging:syslogs
cert = /etc/stunnel/stunnel.pem

Then on my laptop, I installed proxytunnel and configured SSH like that in .ssh/config (based on instructions from https://github.com/proxytunnel/proxytunnel) where the domain name "vps" is also defined in the /etc/hosts file of my laptop and points to the public IPv4 and IPv6 address of my VPS:

Host mytunnel
        ProtocolKeepAlives 30
        ProxyCommand proxytunnel -e -p vps:443 -d:vps:22

I think using the -d parameter should not be needed since sslh would detect the SSH traffic should go to port 22 but proxytunnel needed this parameter.
When I try to connect using the command ssh mytunnel, I can see this error message:

error: connect() failed: [111] Connection refused
kex_exchange_identification: Connection closed by remote host

Do you have an idea what is wrong with my configuration?
By the way, is it necessary to run sslh in inetd mode when using stunnel? On the page https://www.stunnel.org/howto.html, I read "Running in daemon mode is much preferred to running in inetd mode.".
Thanks.

[baptx]

Update: I noticed the sslh / stunnel example uses stunnel3 but the current version is stunnel4 which does not have the "-l", "-d", "-p" and "-f" parameters.

I modified /etc/stunnel/stunnel.conf so it looks like this:

foreground = yes
[sslh]
accept = 443
exec = /usr/sbin/sslh
execArgs = sslh -i --ssh localhost:22
cert = /etc/stunnel/stunnel.pem

And now when I connect with ssh mytunnel on my laptop, I can see another error:

Via vps:443 -> vps:22
error: Socket read error: [104] Connection reset by peer
kex_exchange_identification: Connection closed by remote host

Here is the debug log of the command sudo stunnel on the server:

2020.04.11 17:48:14 LOG5[ui]: stunnel 5.50 on x86_64-pc-linux-gnu platform
2020.04.11 17:48:14 LOG5[ui]: Compiled with OpenSSL 1.1.1b  26 Feb 2019
2020.04.11 17:48:14 LOG5[ui]: Running  with OpenSSL 1.1.1d  10 Sep 2019
2020.04.11 17:48:14 LOG5[ui]: Threading:PTHREAD Sockets:POLL,IPv6,SYSTEMD TLS:ENGINE,FIPS,OCSP,PSK,SNI Auth:LIBWRAP
2020.04.11 17:48:14 LOG5[ui]: Reading configuration from file /etc/stunnel/stunnel.conf
2020.04.11 17:48:14 LOG5[ui]: UTF-8 byte order mark not detected
2020.04.11 17:48:14 LOG5[ui]: FIPS mode disabled
2020.04.11 17:48:14 LOG5[ui]: Configuration successful
2020.04.11 17:48:14 LOG5[ui]: Binding service [sslh] to :::443: Address already in use (98)
2020.04.11 17:48:17 LOG5[0]: Service [sslh] accepted connection from x.x.x.x:50706
2020.04.11 17:48:17 LOG3[0]: SSL_accept: 1408F09B: error:1408F09B:SSL routines:ssl3_get_record:https proxy request
2020.04.11 17:48:17 LOG5[0]: Connection reset: 0 byte(s) sent to TLS, 0 byte(s) sent to socket

I don't know why it says Binding service [sslh] to :::443: Address already in use (98) since I have no other program listening on port 443 and I can see that my laptop was able to connect with stunnel (I replaced my Internet provider IP address with x.x.x.x). My Apache web server is listening on port 4443 instead of 443.

[baptx]

Update 2: it looks like proxytunnel is not needed to use sslh and stunnel. Now sslh and stunnel are working together with the configuration below (based on https://grox.net/sysadm/net/ssh_via_ssl_tunnel.howto) but I cannot get the transparent mode to work when using stunnel and sslh together in order to display the incoming IP address in my Apache web server logs (it only works when using sslh without stunnel).

Here is the file /etc/stunnel/stunnel.conf on the server:

foreground = yes
[sslh]
accept = 443
exec = /usr/sbin/sslh
execArgs = sslh -i --transparent --ssh localhost:22 --http localhost:80
cert = /etc/stunnel/stunnel.pem
transparent = source

I had to comment the transparent option since it blocked the SSH and HTTPS connection to my server. Here is the file /etc/stunnel/stunnel.conf on the client (my laptop, in order to use SSH in an SSL tunnel):

foreground = yes
client = yes
[sslh]
accept = 2222
connect = vps:443

I started stunnel (which uses sslh) on the server with the command sudo stunnel and started stunnel on the client with the command sudo stunnel also. Then I can use SSH in an SSL tunnel with the command ssh localhost -p 2222 on the client.

Do you have an idea what needs to be done in order to see the incoming IP address when using sslh with stunnel?
I am already using the transparent proxy configuration for sslh (https://github.com/yrutschle/sslh/blob/master/doc/config.md#transparent-proxy-support) but adding transparent = source to /etc/stunnel/stunnel.conf on the server blocks SSH and HTTPS connection.
By looking at the documentation of stunnel (https://www.stunnel.org/static/stunnel.html), it seems that it is not needed to add iptables rules if we use transparent = source with the "Local mode (exec option)".
Related issue: #209

[baptx]

Update 3: I have found a way to use sslh and stunnel in transparent mode but for IPv4 only, by replacing this line (from https://github.com/yrutschle/sslh/blob/master/doc/config.md#transparent-proxy-support):
iptables -t nat -A OUTPUT -m owner --uid-owner sslh -p tcp --tcp-flags FIN,SYN,RST,ACK SYN -j CONNMARK --set-xmark 0x01/0x0f
by this line (from https://blog.inf.re/stunnel-transparent-proxy-to-localhost.html https://gist.github.com/Informatic/2034898bc404ea99f322dd0fdfe43878):
iptables -t nat -A OUTPUT -d 127.1.1.1 -p tcp --tcp-flags FIN,SYN,RST,ACK SYN -j CONNMARK --set-xmark 0x01/0x0f
And using this line in the file /etc/stunnel/stunnel.conf on the server:
execArgs = sslh -i --transparent --ssh 127.1.1.1:22 --http 127.1.1.1:80

Is this the best way to do it? Any idea to make IPv6 work in a similar way?
I read that to accept IPv6 with stunnel, we have to use accept :::443 (https://serverfault.com/questions/666712/binding-stunnel-on-both-ipv6-and-ipv4/838259#838259).
With this option, I can see that my web server can be accessed from an IPv6-only subdomain when connecting from an IPv6 address but the transparent mode does not get the correct IPv6 address in the web server logs if I am using both sslh and stunnel.

At the moment, I only managed to get both IPv4 and IPv6 addresses in the web server log if I am using stunnel without sslh (or sslh without stunnel), using this configuration in /etc/stunnel/stunnel.conf:

foreground = yes
[sslh]
accept = vps:443
connect = localhost-stunnel:80
cert = /etc/letsencrypt/live/website/fullchain.pem
key = /etc/letsencrypt/live/website/privkey.pem
transparent = source

In my /etc/hosts file, localhost-stunnel points to 127.1.1.1 and ::2. The local IPv6 address ::2 was added with the command sudo ip -6 address add ::2 dev lo (based on https://serverfault.com/questions/193377/ipv6-loopback-addresses-equivalent-to-127-x-x-x/754287#754287) and I used this command to enable transparent mode for IPv6:
ip6tables -t nat -A OUTPUT -d ::2 -p tcp --tcp-flags FIN,SYN,RST,ACK SYN -j CONNMARK --set-xmark 0x01/0x0f.
If I replace the line connect = localhost-stunnel:80 with these lines to use sslh with stunnel, the web server receives the locahost IPv4 address 127.0.0.1 instead of the remote IPv6 address of the visitor:

exec = /usr/sbin/sslh
execArgs = sslh -i --transparent --ssh localhost-stunnel:22 --http localhost-stunnel:80

Does sslh inetd mode support IPv6 correctly with transparent proxy?

[yrutschle]

I wouldn't expect inetd to work with transparent proxying (inetd would handle the external connection and sslh would only get data from stdin, so basically with inetd I would except sslh to not even know about the client IP). But I could be wrong, I haven't really used inetd much.

On the other hand, sslh should be largely agnostic to the IP version you're using. Whatever works in IPv4 should work the same in IPv4, except maybe for address parsing.

[baptx]

@yrutschle so it is not possible to use sslh with stunnel and keep the original IP address? It looks like there was only a problem with IPv6 when I tested, maybe a software bug in sslh when using inetd mode and transparent proxy.

[yrutschle]

I'm not sure what I said anymore :-) What I say is:

• I don't think using inetd mode would allow you to have transparent networking (with sslh, stunnel, or anything: basically an inetd service would not see the external IP address)
• I don't think sslh should behave differently in IPv6 compared to IPv4. If it does, it's a bug
• sslh in transparent mode definitely works (not in inetd, see above)
• I have no idea if it's possible to string two transparent proxies in a row

[baptx]

@yrutschle thanks for trying to help. If we want to use sslh with stunnel, we have no other choice than using inetd? (the example from your documentation is using inetd)
There is probably a bug since I was able to get inetd working with transparent proxy using IPv4 but not IPv6, do you have an idea where it comes from? (I described step by step what I made in previous comments)