Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

linux branch threading broken #2

Open
BotoX opened this issue Oct 24, 2020 · 4 comments
Open

linux branch threading broken #2

BotoX opened this issue Oct 24, 2020 · 4 comments
Assignees
@agokhale

Comments

@BotoX
Copy link

BotoX commented Oct 24, 2020

Receiver:

$ viamillipede rx 9998
hello

$ viamillipede rx 9998
hello
viamillipede: rx.c:63: rxworker: Assertion `pkt.preamble == preamble_cannon_ul && "preamble check"' failed.
Aborted

$ viamillipede rx 9998
hello

$ viamillipede rx 9998
hello

$ viamillipede rx 9998
hello
viamillipede: rx.c:63: rxworker: Assertion `pkt.preamble == preamble_cannon_ul && "preamble check"' failed.
Aborted

Sender:

$ echo hello | viamillipede tx 10.8.1.1 9998

$ echo hello | viamillipede tx 10.8.1.1 9998

$ echo hello | viamillipede tx 10.8.1.1 9998

$ echo hello | viamillipede tx 10.8.1.1 9998

$ echo hello | viamillipede tx 10.8.1.1 9998

Using the linuxport branch on Linux, compiled with:
gcc -O2 -lpthread plumbing.c tx.c rx.c viamillipede.c terminate.c prbs.c -o viamillipede
(Makefile doesn't work on Linux)

With verbose 20 on the receiver:

$ ./viamillipede rx 9999 verbose 20
verbose set to 20
bind sockid: 3
      listen sockid: 3
rxw:00 threadlaunched
rxw:01 threadlaunched
rxw:02 threadlaunched
rxw:03 threadlaunched
rxw:04 threadlaunched
rxw:05 threadlaunched
rxw:06 threadlaunched
rxw:07 threadlaunched
rxw:08 threadlaunched
rxw:09 threadlaunched
rxw:10 threadlaunched
rxw:11 threadlaunched
rxw:12 threadlaunched
rxw:13 threadlaunched
rxw:14 threadlaunched
rxw:15 threadlaunched
rx: worker group launched
setting input to STDINrxw:05 accepting and locked
   accept sockid: 3
   socket 3 accepted to fd:4 
RCVBUF:20000 SO_SNDBUF:143000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:4
 rxw:05 accepted fd:4 
rxw:05 fd:4 expect yoes
rxw:06 accepting and locked
   accept sockid: 3
   socket 3 accepted to fd:5 
RCVBUF:20000 SO_SNDBUF:143000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:5
 rxw:06 accepted fd:5 
rxw:06 fd:5 expect yoes
rxw:04 accepting and locked
   accept sockid: 3
   socket 3 accepted to fd:6 
RCVBUF:20000 SO_SNDBUF:143000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:6
 rxw:04 accepted fd:6 
rxw:04 fd:6 expect yoes
rxw:07 accepting and locked
   accept sockid: 3
   socket 3 accepted to fd:7 
RCVBUF:20000 SO_SNDBUF:143000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:7
 rxw:07 accepted fd:7 
rxw:07 fd:7 expect yoes
rxw:08 accepting and locked
   accept sockid: 3
   socket 3 accepted to fd:8 
RCVBUF:20000 SO_SNDBUF:143000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:8
 rxw:08 accepted fd:8 
rxw:08 fd:8 expect yoes
rxw:09 accepting and locked
   accept sockid: 3
   socket 3 accepted to fd:9 
RCVBUF:20000 SO_SNDBUF:143000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:9
 rxw:09 accepted fd:9 
rxw:09 fd:9 expect yoes
rxw:03 accepting and locked
   accept sockid: 3
   socket 3 accepted to fd:10 
RCVBUF:20000 SO_SNDBUF:143000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:10
 rxw:03 accepted fd:10 
rxw:03 fd:10 expect yoes
rxw:10 accepting and locked
   accept sockid: 3
   socket 3 accepted to fd:11 
RCVBUF:20000 SO_SNDBUF:143000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:11
 rxw:10 accepted fd:11 
rxw:10 fd:11 expect yoes
rxw:11 accepting and locked
   accept sockid: 3
   socket 3 accepted to fd:12 
RCVBUF:20000 SO_SNDBUF:143000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:12
 rxw:11 accepted fd:12 
rxw:11 fd:12 expect yoes
rxw:12 accepting and locked
   accept sockid: 3
   socket 3 accepted to fd:13 
RCVBUF:20000 SO_SNDBUF:143000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:13
 rxw:12 accepted fd:13 
rxw:12 fd:13 expect yoes
rxw:02 accepting and locked
   accept sockid: 3
   socket 3 accepted to fd:14 
RCVBUF:20000 SO_SNDBUF:143000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:14
 rxw:02 accepted fd:14 
rxw:02 fd:14 expect yoes
rxw:13 accepting and locked
   accept sockid: 3
   socket 3 accepted to fd:15 
RCVBUF:20000 SO_SNDBUF:143000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:15
 rxw:13 accepted fd:15 
rxw:13 fd:15 expect yoes
rxw:14 accepting and locked
   accept sockid: 3
   socket 3 accepted to fd:16 
RCVBUF:20000 SO_SNDBUF:143000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:16
 rxw:14 accepted fd:16 
rxw:14 fd:16 expect yoes
rxw:15 accepting and locked
   accept sockid: 3
   socket 3 accepted to fd:17 
RCVBUF:20000 SO_SNDBUF:143000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:17
 rxw:15 accepted fd:17
rxw:15 fd:17 expect yoes
rxw:01 accepting and locked
   accept sockid: 3
   socket 3 accepted to fd:18
RCVBUF:20000 SO_SNDBUF:143000 SO_SNDLOWAT:1 SO_RCVLOWAT:1
socketfd:18
 rxw:01 accepted fd:18
rxw:01 fd:18 expect yoes
rxw:00 accepting and locked
   accept sockid: 3
   socket 3 accepted to fd:19
RCVBUF:20000 SO_SNDBUF:143000 SO_SNDLOWAT:1 SO_RCVLOWAT:1
socketfd:19
 rxw:00 accepted fd:19
rxw:00 fd:19 expect yoes
rxw:05 leg:0 siz:6 op:cafe caught new leg
rx05 0[0]-[0]

rxw:05 leg:0 buffer filled to :6 wating  on leg:0
initiator oneshot starting initiate: setting output to STDOUTrxw:05 sequenced leg:0[6]
hello
rxw:06 leg:1 siz:0 op:dead caught new leg

rxw:06 leg:1 buffer filled to :0 wating  on leg:0
rxw: writev fd:1 siz:6tx:ingest started on fd:0RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0
socketfd:0
 {t::0} RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0
socketfd:0
 {t::0} RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0
socketfd:0
 {t::0} RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0
socketfd:0
 {t::0} RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0
socketfd:0
 {t::0} RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0
socketfd:0
 {t::0} RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 viamillipede: rx.c:63: rxworker: Assertion `pkt.preamble == preamble_cannon_ul && "preamble check"' failed.
viamillipede: rx.c:63: rxworker: Assertion `pkt.preamble == preamble_cannon_ul && "preamble check"' failed.
viamillipede: rx.c:63: rxworker: Assertion `pkt.preamble == preamble_cannon_ul && "preamble check"' failed.
viamillipede: rx.c:63: rxworker: Assertion `pkt.preamble == preamble_cannon_ul && "preamble check"' failed.
viamillipede: rx.c:63: rxworker: Assertion `pkt.preamble == preamble_cannon_ul && "preamble check"' failed.
viamillipede: rx.c:63: rxworker: Assertion `pkt.preamble == preamble_cannon_ul && "preamble check"' failed.
viamillipede: rx.c:63: rxworker: Assertion `pkt.preamble == preamble_cannon_ul && "preamble check"' failed.
viamillipede: rx.c:63: rxworker: Assertion `pkt.preamble == preamble_cannon_ul && "preamble check"' failed.
viamillipede: rx.c:63: rxworker: Assertion `pkt.preamble == preamble_cannon_ul && "preamble check"' failed.
rxw:05 leg:0 siz:6 op:cafe caught new leg
rxw:05 leg:0 retired due to read len:0 errno:0
rxconf: workers 16, sequencerleg: 1 done: 0
{r:06:a:1:$}    RCVBUF:20000 Aborted (core dumped)

With verbose 20 on the sender, when it happens on the receiver (verbose 20 on the sender side makes the issue happen less frequently)

echo hello | viamillipede tx tatara.botox.bz 9999 verbose 20
verbose set to 20
txw:0 launching tx: chosen target tatara.botox.bz 9999
        connected to tatara.botox.bz:9999
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:3
 txw:0 send checkphrase:yoes
txw:0 expect ok 
txw:0 online fd:3
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:3
 {t:c:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}
RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}

txw:0 idling fd:3
worker buffer allocator: Socket operation on non-socket
checperror txworker_sm:328txw:1 launching tx: chosen target tatara.botox.bz 9999
        connected to tatara.botox.bz:9999
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:4
 txw:1 send checkphrase:yoes
txw:1 expect ok 
txw:1 online fd:4
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:3
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:4
 {t:c:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}
RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}

txw:1 idling fd:4
worker buffer allocator: Socket operation on non-socket
checperror txworker_sm:328txw:2 launching tx: chosen target tatara.botox.bz 9999
        connected to tatara.botox.bz:9999
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:5
 txw:2 send checkphrase:yoes
txw:2 expect ok 
txw:2 online fd:5
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:3
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:4
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:5
 {t:c:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}
RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}

txw:2 idling fd:5
worker buffer allocator: Socket operation on non-socket
checperror txworker_sm:328txw:3 launching tx: chosen target tatara.botox.bz 9999
        connected to tatara.botox.bz:9999
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:6
 txw:3 send checkphrase:yoes
txw:3 expect ok 
txw:3 online fd:6
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:3
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:4
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:5
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:6
 {t:c:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}
RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}

txw:3 idling fd:6
worker buffer allocator: Socket operation on non-socket
checperror txworker_sm:328txw:4 launching tx: chosen target tatara.botox.bz 9999
        connected to tatara.botox.bz:9999
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:7
 txw:4 send checkphrase:yoes
txw:4 expect ok 
txw:4 online fd:7
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:3
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:4
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:5
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:6
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:7
 {t:c:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}
RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}

txw:4 idling fd:7
worker buffer allocator: Socket operation on non-socket
checperror txworker_sm:328txw:5 launching tx: chosen target tatara.botox.bz 9999
        connected to tatara.botox.bz:9999
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:8
 txw:5 send checkphrase:yoes
txw:5 expect ok 
txw:5 online fd:8
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:3
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:4
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:5
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:6
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:7
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:8
 {t:c:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}
RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}

txw:5 idling fd:8
worker buffer allocator: Socket operation on non-socket
checperror txworker_sm:328txw:6 launching tx: chosen target tatara.botox.bz 9999
        connected to tatara.botox.bz:9999
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:9
 txw:6 send checkphrase:yoes
txw:6 expect ok 
txw:6 online fd:9
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:3
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:4
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:5
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:6
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:7
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:8
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:9
 {t:c:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}
RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}

txw:6 idling fd:9
worker buffer allocator: Socket operation on non-socket
checperror txworker_sm:328txw:7 launching tx: chosen target tatara.botox.bz 9999
        connected to tatara.botox.bz:9999
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:10
 txw:7 send checkphrase:yoes
txw:7 expect ok 
txw:7 online fd:10
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:3
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:4
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:5
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:6
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:7
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:8
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:9
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:10
 {t:c:0}
RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}

txw:7 idling fd:10
worker buffer allocator: Socket operation on non-socket
checperror txworker_sm:328txw:8 launching tx: chosen target tatara.botox.bz 9999
        connected to tatara.botox.bz:9999
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:11
 txw:8 send checkphrase:yoes
txw:8 expect ok 
txw:8 online fd:11
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:3
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:4
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:5
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:6
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:7
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:8
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:9
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:10
 {t:i:0}
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:11
 {t:c:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}

txw:8 idling fd:11
worker buffer allocator: Socket operation on non-socket
checperror txworker_sm:328txw:9 launching tx: chosen target tatara.botox.bz 9999
        connected to tatara.botox.bz:9999
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:12
 txw:9 send checkphrase:yoes
txw:9 expect ok 
txw:9 online fd:12
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:3
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:4
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:5
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:6
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:7
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:8
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:9
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:10
 {t:i:0}
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:11
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:12
 {t:c:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}

txw:9 idling fd:12
worker buffer allocator: Socket operation on non-socket
checperror txworker_sm:328txw:10 launching tx: chosen target tatara.botox.bz 9999
        connected to tatara.botox.bz:9999
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:13
 txw:10 send checkphrase:yoes
txw:10 expect ok 
txw:10 online fd:13
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:3
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:4
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:5
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:6
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:7
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:8
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:9
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:10
 {t:i:0}
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:11
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:12
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:13
 {t:c:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}

txw:10 idling fd:13
worker buffer allocator: Socket operation on non-socket
checperror txworker_sm:328txw:11 launching tx: chosen target tatara.botox.bz 9999
        connected to tatara.botox.bz:9999
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:14
 txw:11 send checkphrase:yoes
txw:11 expect ok 
txw:11 online fd:14
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:3
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:4
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:5
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:6
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:7
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:8
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:9
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:10
 {t:i:0}
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:11
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:12
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:13
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:14
 {t:c:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}

txw:11 idling fd:14
worker buffer allocator: Socket operation on non-socket
checperror txworker_sm:328txw:12 launching tx: chosen target tatara.botox.bz 9999
        connected to tatara.botox.bz:9999
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:15
 txw:12 send checkphrase:yoes
txw:12 expect ok 
txw:12 online fd:15
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:3
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:4
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:5
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:6
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:7
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:8
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:9
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:10
 {t:i:0}
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:11
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:12
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:13
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:14
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:15
 {t:c:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}

txw:12 idling fd:15
worker buffer allocator: Socket operation on non-socket
checperror txworker_sm:328txw:13 launching tx: chosen target tatara.botox.bz 9999
        connected to tatara.botox.bz:9999
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:16
 txw:13 send checkphrase:yoes
txw:13 expect ok 
txw:13 online fd:16
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:3
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:4
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:5
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:6
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:7
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:8
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:9
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:10
 {t:i:0}
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:11
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:12
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:13
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:14
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:15
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:16
 {t:c:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}

txw:13 idling fd:16
worker buffer allocator: Socket operation on non-socket
checperror txworker_sm:328txw:14 launching tx: chosen target tatara.botox.bz 9999
        connected to tatara.botox.bz:9999
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:17
 txw:14 send checkphrase:yoes
txw:14 expect ok 
txw:14 online fd:17
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:3
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:4
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:5
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:6
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:7
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:8
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:9
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:10
 {t:i:0}
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:11
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:12
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:13
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:14
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:15
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:16
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:17
 {t:c:0}        RCVBUF:0 SO_SNDBUF:0 SO_SNDLOWAT:0 SO_RCVLOWAT:0 
socketfd:0
 {t:E:0}

txw:14 idling fd:17
worker buffer allocator: Socket operation on non-socket
checperror txworker_sm:328txw:15 launching tx: chosen target tatara.botox.bz 9999
        connected to tatara.botox.bz:9999
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:18
 txw:15 send checkphrase:yoes
txw:15 expect ok 
txw:15 online fd:18
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:3
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:4
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:5
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:6
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:7
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:8
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:9
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:10
 {t:i:0}
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:11
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:12
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:13
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:14
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:15
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:16
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:17
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:18
 {t:c:0}

txw:15 idling fd:18
all tx workers launched RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:3
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:4
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:5
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:6
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:7
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:8
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:9
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:10
 {t:i:0}
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:11
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:12
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:13
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:14
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:15
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:16
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:17
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:18
 {t:i:0}

setting input to STDINinitiator oneshot starting initiate: setting output to STDOUTtx:ingest started on fd:0RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:3
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:4
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:5
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:6
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:7
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:8
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:9
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:10
 {t:i:0}
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:11
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:12
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:13
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:14
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:15
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:16
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:17
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:18
 {t:i:0}


txw:00 read leg 0 : fd:0 reqsiz:200000 rsiz:6
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:3
 {t:d:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:4
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:5
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:6
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:7
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:8
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:9
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:10
 {t:i:0}
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:11
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:12
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:13
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:14
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:15
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:16
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:17
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:18
 {t:i:0}

RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:3
 {t:d:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:4
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:5
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:6
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:7
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:8
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:9
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:10
 {t:i:0}
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:11
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:12
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:13
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:14
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:15
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:16
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:17
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:18
 {t:i:0}

txw:00 psh leg:0.(+2e -6)  aiowrite: Socket operation on non-socket

txw:01 read leg 1 : fd:0 reqsiz:200000 rsiz:0
txingest: stdin exhausted. sending shutdownchecperror txpush:199writesocket: Socket operation on non-socket
checperror txpush:204RCVBUF:200000 txw:09 nothing left to do
SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:3
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:4
 {t:d:1}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:5
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:6
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 txw:00 nothing left to do
SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:7
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:8
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 txw:00 worker doneSO_RCVLOWAT:1 
socketfd:9
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:10
 {t:i:0}
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:11
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 txw:15 nothing left to do
SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:12
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:13
 {t:i:0}        txw:07 nothing left to do
txw:12 nothing left to do
txw:01 nothing left to do
txw:01 psh leg:1.(+28 -0)  RCVBUF:200000 aiowrite: Socket operation on non-socket
checperror txpush:199writesocket: Socket operation on non-socket
checperror txpush:204SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:14
 {t:i:0}        txw:13 nothing left to do
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:15
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:16
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:17
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:18
 {t:i:0}

txw:15 worker donetxw:09 worker donetxw:14 nothing left to do
txw:10 nothing left to do
txw:02 nothing left to do
txw:11 nothing left to do
txw:04 nothing left to do
txw:05 nothing left to do
txw:08 nothing left to do
txw:03 nothing left to do
tx:ingest complete for 6(bytes) in 

RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:3
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:4
 {t:i:1}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:5
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 txw:06 nothing left to do
SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:6
 {t:i:0}        txw:01 worker donetxw:12 worker donetxw:13 worker doneRCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 
socketfd:7
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1
socketfd:8
 {t:i:0}        txw:14 worker doneRCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1
socketfd:9
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1
socketfd:10
 {t:i:0}
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1
socketfd:11
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1
socketfd:12
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1 txw:11 worker donetxw:07 worker donetxw:10 worker done
socketfd:13
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1
socketfd:14
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1
socketfd:15
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1
socketfd:16
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1
socketfd:17
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1
socketfd:18
 {t:i:0}

txw:08 worker donetxw:05 worker donetxw:04 worker donetxw:02 worker donetxw:03 worker donetxw:06 worker done
0(MiBytes)/1(s)=  0.0000(MiBps)
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1
socketfd:3
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1
socketfd:4
 {t:i:1}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1
socketfd:5
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1
socketfd:6
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1
socketfd:7
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1
socketfd:8
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1
socketfd:9
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1
socketfd:10
 {t:i:0}
RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1
socketfd:11
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1
socketfd:12
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1
socketfd:13
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1
socketfd:14
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1
socketfd:15
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1
socketfd:16
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1
socketfd:17
 {t:i:0}        RCVBUF:200000 SO_SNDBUF:200000 SO_SNDLOWAT:1 SO_RCVLOWAT:1
socketfd:18
 {t:i:0}

finished normally```
@BotoX
Copy link
Author

BotoX commented Oct 24, 2020 •
edited
Loading

Issue doesn't seem to happen with:
echo hello | viamillipede tx tatara.botox.bz 9999 threads 1

but multiple threads is the whole point of this, right?

nevermind it still does now that I tried it for a few minutes with zfs send ...
on the sender I get: tx.c:200: txpush: Assertion `txworker->writeremainder + sizeof (struct millipacket_s ) == writeret' failed.
receiver: rx.c:63: rxworker: Assertion `pkt.preamble == preamble_cannon_ul && "preamble check"' failed.

use case: zfs send -c -v -t $TOKEN | mbuffer -s 128k -m 1G | viamillipede tx 10.8.1.1 9999 threads 1
viamillipede rx 9999 | mbuffer -q -s 128k -m 1G | zfs receive -v -s zpool

@agokhale
Copy link
Owner

Thanks for the eyes on the linux branch; I can confirm that threading is fundamentally broken on the linux port ( freebsd is the main target) . And yes; a single transport thread isn't terribly useful. I take a poke at it to see if I can bring the linux branch to parity with freebsd. Also I would try clang as the compiler.

@agokhale agokhale changed the title Broken? linux branch threading broken Oct 24, 2020
@agokhale agokhale self-assigned this Oct 24, 2020
@zxyy-bys
Copy link

@agokhale
For the linux version, you don't need to manually acquire the mutex after pthread_cond_wait(), since the mutex will be locked automatically after the thread waked up from the pthread_cond_wait().

@agokhale
Copy link
Owner

agokhale commented Mar 1, 2021

You are right!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@agokhale agokhale

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@BotoX @agokhale @zxyy-bys