Remote tmux session using :socket leads to wrong tmux path inside session

[aseltmann]

To check the output of my tmux session, I have to attach to the session in a terminal (output directly to org is not yet supported). I noticed that I can not "tmux detach" or "tmux ls" because of tmux version issues. Using C-b d to detach still works. So I can work around it, but I noticed this inconsistency and wanted to report it.
Edit: I found out the reason for this behaviour - it is that the tmux session I attach to itself is created by tmux 3.0a, but the tmux path inside the session is different and so the tmux version is different as well (1.8). Therefore they are not compatible and the active tmux session can not be controlled via shell commands. This only happens with ob-tmux and not with a manually created session - this is what I wanted to report with this issue.

On my local machine I've got (works as expected)

which tmux
tmux -V

#+RESULTS:
| /bin/tmux |      |
| tmux      | 3.0a |

On the remote machine I've got (works as expected):

which tmux
tmux -V

#+RESULTS:
| /usr/local/bin/tmux |      |
| tmux                | 3.0a |

If I open a tmux session from my terminal and start a new session with tmux new -d and then attach to it I get (works as expected):

[user@remote ~]$ which tmux
/usr/local/bin/tmux
[user@remote ~]$ tmux -V
tmux 3.0a

If I use ob-tmux to open a tmux session following the README where in the end I have a code block somewhat like this #+BEGIN_SRC tmux :socket ~/.tmux-local-socket-remote-machine :session tmuxtest and then attach to it from my terminal I get (does not work as expected):

[user@remote ~]$ which tmux
/bin/tmux
[user@remote ~]$ tmux -V
tmux 1.8

I already tried to specify the desired path /usr/local/bin/tmux in org-babel-tmux-location (and soft- and hardlinked tmux on my own machine to this place, since otherwise ob-tmux did not create a session. The result was still the one you see above.

Here are some details on my setup, please tell me if you need more:

GNU Emacs 26.3

Org mode version 9.1.9 (release_9.1.9-65-g5e4542 @ /usr/share/emacs/26.3/lisp/org/)

(use-package ob-tmux
 ;; Install package automatically (optional)
 :ensure t
 :custom
 (org-babel-default-header-args:tmux
  '((:results . "silent")	;
    (:session . "default")	; The default tmux session to send code to
    (:socket  . nil)))		; The default tmux socket to communicate with
 ;; The tmux sessions are prefixed with the following string.
 ;; You can customize this if you like.
 (org-babel-tmux-session-prefix "ob-")
 ;; The terminal that will be used.
 ;; You can also customize the options passed to the terminal.
 ;; The default terminal is "gnome-terminal" with options "--".
 (org-babel-tmux-terminal "xterm")
 (org-babel-tmux-terminal-opts '("-T" "ob-tmux" "-e"))
 ;; Finally, if your tmux is not in your $PATH for whatever reason, you
 ;; may set the path to the tmux binary as follows:
 (org-babel-tmux-location "/usr/local/bin/tmux"))

[ahendriksen]

Hi,
Thanks for reporting this issue! It sure looks complicated. I will try to look into it, but I might not be able to help you..

If I understand correctly, you can use tmux detach when you manually login to the remote and attach to the tmux session, but this fails when you execute it from an ob-tmux source block?

[aseltmann]

tmux detach not working was just an illustration that inside the tmux session, which is created through ob-tmux, the tmux version is different than outside.

So on this remote machine I log in to there are two tmux versions: a global one under /bin/tmux with version 1.8 and a local one under /usr/local/bin/tmux with version 3.0a. I need 3.0a.

• If I attach to a manually created session, inside this session the tmux path is still /usr/local/bin/tmux with version 3.0a
• If I attach to a session created by ob-tmux, inside this session the tmux path is suddenly /bin/tmux with version 1.8

In my first comment I added some remarks pointing out what worked as intended and what did not work.

[ahendriksen]

Hi,
Thanks for the clarifying comment. I believe I understand the problem now :)

I tried to reproduce the problem for myself by first creating a socket forwarded socket:

export HOST=<remote> ;
ssh $HOST -tfN -L ~/.ssh/.tmux-$HOST:$(ssh $HOST  'tmux ls -F "#{socket_path}"' | head -1)

And by creating a new session from the socket:

tmux -S ~/.ssh/.tmux-$HOST  new-session -d -s "session-from-socket" -n "0" 

Then by creating a new session on the remote:

ssh $HOST
tmux new-session -d -s "session-from-remote" -n "0"

After attaching, If I compare

env | sort

on the two sessions, I only see minor differences (DISPLAY is defined in one of the sessions, but not on the other for instance)

I get the same result for

tmux -V

on both sessions. I have versions 2.8 locally and 2.3 remotely.

So it could be that your tmux versions are just incompatible. Have you tried comparing the socket creation with the two different tmuxes? ie:

export HOST=<remote> ;
ssh $HOST -tfN -L ~/.ssh/.tmux-$HOST:$(ssh $HOST  '/bin/tmux ls -F "#{socket_path}"' | head -1)

or

export HOST=<remote> ;
ssh $HOST -tfN -L ~/.ssh/.tmux-$HOST:$(ssh $HOST  '/usr/local/bin/tmux ls -F "#{socket_path}"' | head -1)

Perhaps this is the culprit?

[aseltmann]

I finally found some time to try this out:

	ssh $HOST; tmux new-session -d -s "session-from-remote" -n "0"	tmux -S ~/.ssh/.tmux-$HOST new-session -d -s "session-from-socket" -n "0"
	tmux attach -t session-from-remote	tmux attach -t session-from-socket
which tmux	/usr/local/bin/tmux	/bin/tmux
tmux -V	tmux 3.0a	tmux 1.8
env | sort	}	}
	BASH_FUNC_module()=() { eval `/usr/bin/modulecmd bash $*`	BASH_FUNC_module()=() { eval `/usr/bin/modulecmd bash $*`
		_=/bin/env
		DISPLAY=:0
	HISTCONTROL=ignoredups	HISTCONTROL=ignoredups
	HISTSIZE=1000	HISTSIZE=1000
	HOME=/home/USER2²	HOME=/home/USER2²
	HOSTNAME=…	HOSTNAME=…
	http_proxy=…	http_proxy=…
	https_proxy=…	https_proxy=…
	LANG=en_US.UTF-8	LANG=en_US.UTF-8
	LESSOPEN=
	LOADEDMODULES=	LOADEDMODULES=
	LOGNAM=USER2²	LOGNAM=USER2²
	LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:.tar=01;31:.tgz=01;31:.arc=01;31:.arj=01;31:.taz=01;31:.lha=01;31:.lz4=01;31:.lzh=01;31:.lzma=01;31:.tlz=01;31:.txz=01;31:.tzo=01;31:.t7z=01;31:.zip=01;31:.z=01;31:.Z=01;31:.dz=01;31:.gz=01;31:.lrz=01;31:.lz=01;31:.lzo=01;31:.xz=01;31:.bz2=01;31:.bz=01;31:.tbz=01;31:.tbz2=01;31:.tz=01;31:.deb=01;31:.rpm=01;31:.jar=01;31:.war=01;31:.ear=01;31:.sar=01;31:.rar=01;31:.alz=01;31:.ace=01;31:.zoo=01;31:.cpio=01;31:.7z=01;31:.rz=01;31:.cab=01;31:.jpg=01;35:.jpeg=01;35:.gif=01;35:.bmp=01;35:.pbm=01;35:.pgm=01;35:.ppm=01;35:.tga=01;35:.xbm=01;35:.xpm=01;35:.tif=01;35:.tiff=01;35:.png=01;35:.svg=01;35:.svgz=01;35:.mng=01;35:.pcx=01;35:.mov=01;35:.mpg=01;35:.mpeg=01;35:.m2v=01;35:.mkv=01;35:.webm=01;35:.ogm=01;35:.mp4=01;35:.m4v=01;35:.mp4v=01;35:.vob=01;35:.qt=01;35:.nuv=01;35:.wmv=01;35:.asf=01;35:.rm=01;35:.rmvb=01;35:.flc=01;35:.avi=01;35:.fli=01;35:.flv=01;35:.gl=01;35:.dl=01;35:.xcf=01;35:.xwd=01;35:.yuv=01;35:.cgm=01;35:.emf=01;35:.axv=01;35:.anx=01;35:.ogv=01;35:.ogx=01;35:.aac=01;36:.au=01;36:.flac=01;36:.mid=01;36:.midi=01;36:.mka=01;36:.mp3=01;36:.mpc=01;36:.ogg=01;36:.ra=01;36:.wav=01;36:.axa=01;36:.oga=01;36:.spx=01;36:*.xspf=01;36:	LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:.tar=01;31:.tgz=01;31:.arc=01;31:.arj=01;31:.taz=01;31:.lha=01;31:.lz4=01;31:.lzh=01;31:.lzma=01;31:.tlz=01;31:.txz=01;31:.tzo=01;31:.t7z=01;31:.zip=01;31:.z=01;31:.Z=01;31:.dz=01;31:.gz=01;31:.lrz=01;31:.lz=01;31:.lzo=01;31:.xz=01;31:.bz2=01;31:.bz=01;31:.tbz=01;31:.tbz2=01;31:.tz=01;31:.deb=01;31:.rpm=01;31:.jar=01;31:.war=01;31:.ear=01;31:.sar=01;31:.rar=01;31:.alz=01;31:.ace=01;31:.zoo=01;31:.cpio=01;31:.7z=01;31:.rz=01;31:.cab=01;31:.jpg=01;35:.jpeg=01;35:.gif=01;35:.bmp=01;35:.pbm=01;35:.pgm=01;35:.ppm=01;35:.tga=01;35:.xbm=01;35:.xpm=01;35:.tif=01;35:.tiff=01;35:.png=01;35:.svg=01;35:.svgz=01;35:.mng=01;35:.pcx=01;35:.mov=01;35:.mpg=01;35:.mpeg=01;35:.m2v=01;35:.mkv=01;35:.webm=01;35:.ogm=01;35:.mp4=01;35:.m4v=01;35:.mp4v=01;35:.vob=01;35:.qt=01;35:.nuv=01;35:.wmv=01;35:.asf=01;35:.rm=01;35:.rmvb=01;35:.flc=01;35:.avi=01;35:.fli=01;35:.flv=01;35:.gl=01;35:.dl=01;35:.xcf=01;35:.xwd=01;35:.yuv=01;35:.cgm=01;35:.emf=01;35:.axv=01;35:.anx=01;35:.ogv=01;35:.ogx=01;35:.aac=01;36:.au=01;36:.flac=01;36:.mid=01;36:.midi=01;36:.mka=01;36:.mp3=01;36:.mpc=01;36:.ogg=01;36:.ra=01;36:.wav=01;36:.axa=01;36:.oga=01;36:.spx=01;36:*.xspf=01;36:
	MAIL=…	MAIL=…
	MATHEMATICA_HOME=/cluster/apps/mathematica/11.3	MATHEMATICA_HOME=/cluster/apps/mathematica/11.3
	MODULEPATH=/usr/share/Modules/modulefiles:/etc/modulefiles:/cluster/modulefiles	MODULEPATH=/usr/share/Modules/modulefiles:/etc/modulefiles:/cluster/modulefiles
	MODULESHOME=/usr/share/Modules	MODULESHOME=/usr/share/Modules
	PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/USER2²/.local/bin:/home/USER2²/bin:/home/USER2²/.local/bin:/home/USER²/bin	PATH=/home/USER¹/path/to/miniconda3/bin:/home/USER¹/path/to/miniconda3/condabin:/home/USER¹/.local/bin:/bin:/usr/bin:/usr/local/bin:/usr/local/sbin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/var/lib/snapd/snap/bin:/home/USER¹/path/to/miniconda3/bin:/usr/sbin:/home/USER2²/.local/bin:/home/USER2²/bin
	PWD=/home/USER2²	PWD=/home/USER2²
	QTDIR=/usr/lib64/qt-3.3	QTDIR=/usr/lib64/qt-3.3
	QT_GRAPHICSSYSTEM_CHECKED=1	QT_GRAPHICSSYSTEM_CHECKED=1
	QTINC=/usr/lib64/qt-3.3/include	QTINC=/usr/lib64/qt-3.3/include
	QTLIB=/usr/lib64/qt-3.3/lib	QTLIB=/usr/lib64/qt-3.3/lib
	SHELL=/bin/bash	SHELL=/bin/bash
	SHLVL=2	SHLVL=2
	SSH_CLIENT=…	SSH_CLIENT=…
	SSH_CONNECTION=…
	SSH_TTY=/dev/pts/60	SSH_TTY=/dev/pts/60
	TERM=screen	TERM=screen
	TMUX_PANE=%9	TMUX_PANE=%8
	TMUX=/tmp/tmux-67339/default,26721,6	TMUX=/tmp/tmux-67339/default,26721,5
	USER=USER2²	USER=USER2²
	_=/usr/bin/env
		WINDOWID=41943047
		XAUTHORITY=/tmp/xauth-1000-_0
	XDG_RUNTIME_DIR=/run/user/67339	XDG_RUNTIME_DIR=/run/user/67339
	XDG_SESSION_ID=2202	XDG_SESSION_ID=2202

¹ USER: my username at the local machine
² USER2: my username at the remote machine

As you can see there are bigger differences in env | sort, I think especially PATH is relevant, but some other fields are different as well. Maybe it is important to stress, that I think the problem arises from 2 different tmux versions at the remote (which is a common thing for multi-user server setups, I think).

So it could be that your tmux versions are just incompatible. Have you tried comparing the socket creation with the two different tmuxes? ie:

• ...ssh $HOST '/bin/tmux ls -F...: After logging into the first SSH, this fails with the following message (the second SSH is not getting prompted):

  protocol version mismatch (client 7, server 8)
  Bad local forwarding specification '/home/USER/.ssh/.tmux-HOST:'
  

• ...ssh $HOST '/usr/local/bin/tmux ls -F...: Works. Same result (see tables above) as ssh $HOST 'tmux ls -F.

Since the socket creation is only possible with the second version, and the same problem persists, this does not seem to be the culprit.

EDIT: Today when using the output function discussed in issue #6 I experienced something new: I could send the command to the tmux remote server (verified via direct ssh control over command line), BUT after calling org-babel-open-src-block-result in the org-edit-special buffer I just got:

#+RESULTS:
: server version is too old for client

After I did (setq org-babel-tmux-location "/usr/local/bin/tmux") it worked again as before.