Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Assertion failed: (pkt.preamble == preamble_cannon_ul && "preamble check"), function rxworker, file rx.c, line 63. #3

Open
dch opened this issue Jan 20, 2021 · 12 comments
Open

Assertion failed: (pkt.preamble == preamble_cannon_ul && "preamble check"), function rxworker, file rx.c, line 63. #3

dch opened this issue Jan 20, 2021 · 12 comments

Comments

@dch
Copy link

dch commented Jan 20, 2021 •
edited
Loading

v0.89 built from ports, FreeBSD 12.2R amd64 to FreeBSD 13.0-ALPHA1 amd64.

# receiver
$ time viamillipede rx 7000 > /dev/null
...
Assertion failed: (pkt.preamble == preamble_cannon_ul && "preamble check"), function rxworker, file rx.c, line 63.

# sender
$ host local.skunkwerks.at
local.skunkwerks.at has address 94.136.7.161
$ sudo zfs send -LevcpDR zroot@20210120-1307 |  viamillipede tx local.skunkwerks.at 7000
# hangs after receiver blows out
...
@dch
Copy link
Author

dch commented Jan 20, 2021

seems to be reproducible here, after only a few seconds I will try with tar at both ends so we can see what file its on when it crashes.

Would pcap be helpful?

@dch
Copy link
Author

dch commented Jan 20, 2021

when I use tar at both ends, this seems to be stable for much longer, maybe this only occurs with /dev/null sink?

@agokhale
Copy link
Owner

The preamble check means that the very start of a tcp worker, I'm not sure about the devnull endpoint changing anything

  1. what does ^T report ( both tx and rx part should respond 0
  2. can your repro survive 'verbose 6' while running?

If in doubt, core it and I'll sift through it.

@dch
Copy link
Author

dch commented Jan 20, 2021 

# verbose 6 rx
           
 being a server at port 7000 

 rxw:00 accepted fd:5 
rxw:01 accepted fd:6 
rxw:02 accepted fd:7 
rxw:03 accepted fd:8 
rxw:04 accepted fd:9 
rxw:05 accepted fd:10 
rxw:06 accepted fd:11 
rxw:07 accepted fd:12 
rxw:08 accepted fd:13 
rxw:09 accepted fd:14 
rxw:10 accepted fd:15 
rxw:11 accepted fd:16 
rxw:12 accepted fd:17 
rxw:13 accepted fd:18 
rxw:14 accepted fd:19 
rxw:15 accepted fd:20 
{t::0}	{t::0}	{t::0}	{t::0}	{t::0}	{t::0}	{t::0}	{t::0}	
{t::0}	{t::0}	{t::0}	{t::0}	{t::0}	{t::0}	{t::0}	{t::0}	

Assertion failed: (pkt.preamble == preamble_cannon_ul && "preamble check"), function rxworker, file rx.c, line 63.

________________________________________________________
Executed in   82.67 secs   fish           external 
   usr time    0.82 secs    0.00 millis    0.82 secs 
   sys time    2.44 secs    5.80 millis    2.43 secs 

fish: Process 48429, “viamillipede” “time viamillipede verbose 6 rx…” terminated by signal SIGABRT (Abort)

the log below isn't 100% reliable (tmux scrollback) but is probably still useful:

# verbose 6 tx
$ sudo zfs send -LecpDR zroot@20210120-1307 |  viamillipede verbose 6 tx local.skunkwerks.at 7000
tx host: local.skunkwerks.at port:7000 
tx: chosen target local.skunkwerks.at 7000
tx: chosen target local.skunkwerks.at 7000
tx: chosen target local.skunkwerks.at 7000
tx: chosen target local.skunkwerks.at 7000
tx: chosen target local.skunkwerks.at 7000
tx: chosen target local.skunkwerks.at 7000
tx: chosen target local.skunkwerks.at 7000
tx: chosen target local.skunkwerks.at 7000
tx: chosen target local.skunkwerks.at 7000
tx: chosen target local.skunkwerks.at 7000
tx: chosen target local.skunkwerks.at 7000
tx: chosen target local.skunkwerks.at 7000
tx: chosen target local.skunkwerks.at 7000
tx: chosen target local.skunkwerks.at 7000
tx: chosen target local.skunkwerks.at 7000
tx: chosen target local.skunkwerks.at 7000

{t:P:0} {t:P:1} {t:d:2} {t:i:0} {t:i:0} {t:i:0} {t:i:0} {t:i:0}
{t:i:0} {t:i:0} {t:i:0} {t:i:0} {t:i:0} {t:i:0} {t:i:0} {t:i:0}

{t:P:0} {t:P:1} {t:P:2} {t:d:3} {t:i:0} {t:i:0} {t:i:0} {t:i:0}
{t:i:0} {t:i:0} {t:i:0} {t:i:0} {t:i:0} {t:i:0} {t:i:0} {t:i:0}

{t:P:0} {t:P:1} {t:P:2} {t:P:3} {t:d:4} {t:i:0} {t:i:0} {t:i:0}
{t:i:0} {t:i:0} {t:i:0} {t:i:0} {t:i:0} {t:i:0} {t:i:0} {t:i:0}

{t:P:0} {t:P:1} {t:P:2} {t:P:3} {t:P:4} {t:d:5} {t:i:0} {t:i:0}
{t:i:0} {t:i:0} {t:i:0} {t:i:0} {t:i:0} {t:i:0} {t:i:0} {t:i:0}

{t:P:0} {t:P:1} {t:P:2} {t:P:3} {t:P:4} {t:P:5} {t:d:6} {t:i:0}
{t:i:0} {t:i:0} {t:i:0} {t:i:0} {t:i:0} {t:i:0} {t:i:0} {t:i:0}

{t:P:0} {t:P:1} {t:P:2} {t:P:3} {t:P:4} {t:P:5} {t:P:6} {t:d:7}
{t:i:0} {t:i:0} {t:i:0} {t:i:0} {t:i:0} {t:i:0} {t:i:0} {t:i:0}

{t:P:0} {t:P:1} {t:P:2} {t:P:3} {t:P:4} {t:P:5} {t:P:6} {t:P:7}
{t:d:8} {t:i:0} {t:i:0} {t:i:0} {t:i:0} {t:i:0} {t:i:0} {t:i:0}

{t:P:0} {t:P:1} {t:P:2} {t:P:3} {t:P:4} {t:P:5} {t:P:6} {t:P:7}
{t:P:8} {t:d:9} {t:i:0} {t:i:0} {t:i:0} {t:i:0} {t:i:0} {t:i:0}

{t:P:0} {t:P:1} {t:P:2} {t:P:3} {t:P:4} {t:P:5} {t:P:6} {t:P:7}
{t:P:8} {t:P:9} {t:d:a} {t:i:0} {t:i:0} {t:i:0} {t:i:0} {t:i:0}

{t:P:0} {t:P:1} {t:P:2} {t:P:3} {t:P:4} {t:P:5} {t:P:6} {t:P:7}
{t:P:8} {t:P:9} {t:P:a} {t:d:b} {t:i:0} {t:i:0} {t:i:0} {t:i:0}

{t:P:0} {t:P:1} {t:P:2} {t:P:3} {t:P:4} {t:P:5} {t:P:6} {t:P:7}
{t:P:8} {t:P:9} {t:P:a} {t:P:b} {t:d:c} {t:i:0} {t:i:0} {t:i:0}

{t:P:0} {t:P:1} {t:P:2} {t:P:3} {t:P:4} {t:P:5} {t:P:6} {t:P:7}
{t:P:8} {t:P:9} {t:P:a} {t:P:b} {t:P:c} {t:d:d} {t:i:0} {t:i:0}

{t:P:0} {t:P:1} {t:P:2} {t:P:3} {t:P:4} {t:P:5} {t:P:6} {t:P:7}
{t:P:8} {t:P:9} {t:P:a} {t:P:b} {t:P:c} {t:P:d} {t:d:e} {t:i:0}

{t:P:0} {t:P:1} {t:P:2} {t:P:3} {t:P:4} {t:P:5} {t:P:6} {t:P:7}
{t:P:8} {t:P:9} {t:P:a} {t:P:b} {t:P:c} {t:P:d} {t:P:e} {t:d:f}

{t:d:10}        {t:P:1} {t:P:2} {t:P:3} {t:P:4} {t:P:5} {t:P:6} {t:P:7}
{t:P:8} {t:P:9} {t:P:a} {t:P:b} {t:P:c} {t:P:d} {t:P:e} {t:P:f}

{t:P:10}        {t:d:11}        {t:P:2} {t:P:3} {t:P:4} {t:P:5} {t:P:6} {t:P:7}
{t:P:8} {t:P:9} {t:P:a} {t:P:b} {t:P:c} {t:P:d} {t:P:e} {t:P:f}

{t:P:10}        {t:P:11}        {t:d:12}        {t:P:3} {t:P:4} {t:P:5} {t:P:6} {t:P:7}
{t:P:8} {t:P:9} {t:P:a} {t:P:b} {t:P:c} {t:P:d} {t:P:e} {t:P:f}

{t:P:10}        {t:P:11}        {t:P:12}        {t:d:13}        {t:P:4} {t:i:5} {t:P:6} {t:P:7}
{t:P:8} {t:P:9} {t:P:a} {t:P:b} {t:P:c} {t:P:d} {t:P:e} {t:P:f}

{t:P:10}        {t:P:11}        {t:P:12}        {t:P:13}        {t:P:4} {t:d:14}        {t:P:6} {t:P:7}
{t:P:8} {t:P:9} {t:P:a} {t:P:b} {t:P:c} {t:P:d} {t:P:e} {t:P:f}

{t:P:10}        {t:P:11}        {t:P:12}        {t:P:13}        {t:P:4} {t:P:14}        {t:P:15}        {t:P:7}
{t:P:8} {t:P:9} {t:P:a} {t:P:b} {t:P:c} {t:P:d} {t:P:e} {t:P:f}

{t:P:10}        {t:P:11}        {t:P:12}        {t:P:13}        {t:d:16}        {t:P:14}        {t:P:15}        {t:P:7}
{t:P:8} {t:P:9} {t:P:a} {t:P:b} {t:P:c} {t:P:d} {t:P:e} {t:P:f}

{t:P:10}        {t:P:11}        {t:P:12}        {t:P:13}        {t:P:16}        {t:P:14}        {t:P:15}        {t:P:7}
{t:d:17}        {t:P:9} {t:P:a} {t:P:b} {t:P:c} {t:P:d} {t:P:e} {t:P:f}

{t:P:10}        {t:P:11}        {t:P:12}        {t:P:13}        {t:P:16}        {t:P:14}        {t:P:15}        {t:d:18}
{t:P:17}        {t:P:9} {t:P:a} {t:P:b} {t:P:c} {t:P:d} {t:P:e} {t:P:f}

{t:P:10}        {t:P:11}        {t:P:12}        {t:P:13}        {t:P:16}        {t:P:14}        {t:P:15}        {t:P:18}
{t:P:17}        {t:P:9} {t:P:19}        {t:P:b} {t:P:c} {t:P:d} {t:P:e} {t:P:f}

{t:P:10}        {t:P:11}        {t:P:12}        {t:P:13}        {t:P:16}        {t:P:14}        {t:P:15}        {t:P:18}
{t:P:17}        {t:d:1a}        {t:P:19}        {t:P:b} {t:P:c} {t:P:d} {t:P:e} {t:P:f}

{t:P:10}        {t:P:11}        {t:P:12}        {t:P:13}        {t:P:16}        {t:P:14}        {t:P:15}        {t:P:18}
{t:P:17}        {t:P:1a}        {t:P:19}        {t:d:1b}        {t:P:c} {t:P:d} {t:P:e} {t:P:f}

{t:P:10}        {t:P:11}        {t:P:12}        {t:P:13}        {t:P:16}        {t:P:14}        {t:P:15}        {t:P:18}
{t:P:17}        {t:P:1a}        {t:P:19}        {t:P:1b}        {t:i:c} {t:P:d} {t:P:e} {t:d:1c}

{t:P:10}        {t:P:11}        {t:P:12}        {t:P:13}        {t:P:16}        {t:P:14}        {t:P:15}        {t:P:18}
{t:P:17}        {t:P:1a}        {t:P:19}        {t:P:1b}        {t:d:1d}        {t:i:d} {t:P:e} {t:P:1c}

{t:P:10}        {t:P:11}        {t:P:12}        {t:P:13}        {t:P:16}        {t:P:14}        {t:P:15}        {t:P:18}
{t:P:17}        {t:P:1a}        {t:P:19}        {t:P:1b}        {t:P:1d}        {t:d:1e}        {t:P:e} {t:P:1c}

{t:P:10}        {t:P:11}        {t:P:12}        {t:P:13}        {t:P:16}        {t:P:14}        {t:P:15}        {t:P:18}
{t:P:17}        {t:P:1a}        {t:P:19}        {t:P:1b}        {t:P:1d}        {t:P:1e}        {t:P:1f}        {t:P:1c}

{t:d:20}        {t:P:11}        {t:P:12}        {t:P:13}        {t:P:16}        {t:P:14}        {t:P:15}        {t:P:18}
{t:P:17}        {t:P:1a}        {t:P:19}        {t:P:1b}        {t:P:1d}        {t:P:1e}        {t:P:1f}        {t:P:1c}

{t:P:20}        {t:d:21}        {t:P:12}        {t:P:13}        {t:P:16}        {t:P:14}        {t:P:15}        {t:P:18}
{t:P:17}        {t:P:1a}        {t:P:19}        {t:P:1b}        {t:P:1d}        {t:P:1e}        {t:P:1f}        {t:P:1c}

{t:P:20}        {t:P:21}        {t:P:12}        {t:P:13}        {t:P:16}        {t:d:22}        {t:P:15}        {t:P:18}
{t:P:17}        {t:P:1a}        {t:P:19}        {t:P:1b}        {t:P:1d}        {t:P:1e}        {t:P:1f}        {t:P:1c}

{t:P:20}        {t:P:21}        {t:P:12}        {t:d:23}        {t:P:16}        {t:P:22}        {t:P:15}        {t:P:18}
{t:P:17}        {t:P:1a}        {t:P:19}        {t:P:1b}        {t:P:1d}        {t:P:1e}        {t:P:1f}        {t:P:1c}

{t:P:20}        {t:P:21}        {t:P:12}        {t:P:23}        {t:P:16}        {t:P:22}        {t:d:24}        {t:P:18}
{t:P:17}        {t:P:1a}        {t:P:19}        {t:P:1b}        {t:P:1d}        {t:P:1e}        {t:P:1f}        {t:P:1c}

{t:P:20}        {t:P:21}        {t:d:25}        {t:P:23}        {t:P:16}        {t:P:22}        {t:P:24}        {t:P:18}
{t:P:17}        {t:P:1a}        {t:P:19}        {t:P:1b}        {t:P:1d}        {t:P:1e}        {t:P:1f}        {t:P:1c}

{t:P:20}        {t:P:21}        {t:P:25}        {t:P:23}        {t:d:26}        {t:P:22}        {t:P:24}        {t:P:18}
{t:P:17}        {t:P:1a}        {t:P:19}        {t:P:1b}        {t:P:1d}        {t:P:1e}        {t:P:1f}        {t:P:1c}

{t:P:20}        {t:P:21}        {t:P:25}        {t:P:23}        {t:P:26}        {t:P:22}        {t:P:24}        {t:P:18}
{t:d:27}        {t:P:1a}        {t:P:19}        {t:P:1b}        {t:P:1d}        {t:P:1e}        {t:P:1f}        {t:P:1c}

{t:P:20}        {t:P:21}        {t:P:25}        {t:P:23}        {t:P:26}        {t:P:22}        {t:P:24}        {t:d:28}
{t:P:27}        {t:P:1a}        {t:P:19}        {t:P:1b}        {t:P:1d}        {t:P:1e}        {t:P:1f}        {t:P:1c}

{t:P:20}        {t:P:21}        {t:P:25}        {t:P:23}        {t:P:26}        {t:P:22}        {t:P:24}        {t:P:28}
{t:P:27}        {t:P:1a}        {t:d:29}        {t:P:1b}        {t:P:1d}        {t:P:1e}        {t:P:1f}        {t:P:1c}

{t:P:20}        {t:P:21}        {t:P:25}        {t:P:23}        {t:P:26}        {t:P:22}        {t:P:24}        {t:P:28}
{t:P:27}        {t:d:2a}        {t:P:29}        {t:P:1b}        {t:P:1d}        {t:P:1e}        {t:P:1f}        {t:P:1c}

{t:P:20}        {t:P:21}        {t:P:25}        {t:P:23}        {t:P:26}        {t:P:22}        {t:P:24}        {t:P:28}
{t:P:27}        {t:P:2a}        {t:P:29}        {t:d:2b}        {t:P:1d}        {t:P:1e}        {t:P:1f}        {t:P:1c}

{t:P:20}        {t:P:21}        {t:P:25}        {t:P:23}        {t:P:26}        {t:P:22}        {t:P:24}        {t:P:28}
{t:P:27}        {t:P:2a}        {t:P:29}        {t:P:2b}        {t:P:1d}        {t:P:1e}        {t:P:1f}        {t:d:2c}

{t:P:20}        {t:P:21}        {t:P:25}        {t:P:23}        {t:P:26}        {t:P:22}        {t:P:24}        {t:P:28}
{t:P:27}        {t:P:2a}        {t:P:29}        {t:P:2b}        {t:P:1d}        {t:P:2d}        {t:P:1f}        {t:P:2c}

{t:P:20}        {t:P:21}        {t:P:25}        {t:P:23}        {t:P:26}        {t:P:22}        {t:P:24}        {t:P:28}
{t:P:27}        {t:P:2a}        {t:P:29}        {t:P:2b}        {t:d:2e}        {t:P:2d}        {t:P:1f}        {t:P:2c}

{t:P:20}        {t:P:21}        {t:P:25}        {t:P:23}        {t:P:26}        {t:P:22}        {t:P:24}        {t:P:28}
{t:P:27}        {t:P:2a}        {t:P:29}        {t:P:2b}        {t:P:2e}        {t:P:2d}        {t:P:2f}        {t:P:2c}

{t:P:20}        {t:P:21}        {t:P:25}        {t:P:23}        {t:P:26}        {t:P:22}        {t:P:24}        {t:d:30}
{t:P:27}        {t:P:2a}        {t:P:29}        {t:P:2b}        {t:P:2e}        {t:P:2d}        {t:P:2f}        {t:P:2c}

{t:P:20}        {t:P:21}        {t:P:25}        {t:P:23}        {t:P:26}        {t:P:22}        {t:P:31}        {t:P:30}
{t:P:27}        {t:P:2a}        {t:P:29}        {t:P:2b}        {t:P:2e}        {t:P:2d}        {t:P:2f}        {t:P:2c}

{t:P:20}        {t:P:21}        {t:P:25}        {t:P:23}        {t:d:32}        {t:P:22}        {t:P:31}        {t:P:30}
{t:P:27}        {t:P:2a}        {t:P:29}        {t:P:2b}        {t:P:2e}        {t:P:2d}        {t:P:2f}        {t:P:2c}

{t:d:33}        {t:P:21}        {t:P:25}        {t:P:23}        {t:P:32}        {t:P:22}        {t:P:31}        {t:P:30}
{t:P:27}        {t:P:2a}        {t:P:29}        {t:P:2b}        {t:P:2e}        {t:P:2d}        {t:P:2f}        {t:P:2c}

{t:P:33}        {t:P:21}        {t:P:25}        {t:d:34}        {t:P:32}        {t:P:22}        {t:P:31}        {t:P:30}
{t:P:27}        {t:P:2a}        {t:P:29}        {t:P:2b}        {t:P:2e}        {t:P:2d}        {t:P:2f}        {t:P:2c}

{t:P:33}        {t:P:21}        {t:P:25}        {t:P:34}        {t:P:32}        {t:d:35}        {t:P:31}        {t:P:30}
{t:P:27}        {t:P:2a}        {t:P:29}        {t:P:2b}        {t:P:2e}        {t:P:2d}        {t:P:2f}        {t:P:2c}

{t:P:33}        {t:P:21}        {t:d:36}        {t:P:34}        {t:P:32}        {t:P:35}        {t:P:31}        {t:P:30}
{t:P:27}        {t:P:2a}        {t:P:29}        {t:P:2b}        {t:P:2e}        {t:P:2d}        {t:P:2f}        {t:P:2c}

{t:P:33}        {t:P:21}        {t:P:36}        {t:P:34}        {t:P:32}        {t:P:35}        {t:P:31}        {t:P:30}
{t:P:27}        {t:P:2a}        {t:P:29}        {t:P:2b}        {t:P:2e}        {t:P:2d}        {t:d:37}        {t:P:2c}

{t:P:33}        {t:P:21}        {t:P:36}        {t:P:34}        {t:P:32}        {t:P:35}        {t:P:31}        {t:P:30}
{t:P:27}        {t:P:38}        {t:P:29}        {t:P:2b}        {t:P:2e}        {t:P:2d}        {t:P:37}        {t:P:2c}

{t:P:33}        {t:P:21}        {t:P:36}        {t:P:34}        {t:P:32}        {t:P:35}        {t:P:31}        {t:P:30}
{t:P:27}        {t:P:38}        {t:P:29}        {t:d:39}        {t:P:2e}        {t:P:2d}        {t:P:37}        {t:P:2c}

{t:P:33}        {t:P:21}        {t:P:36}        {t:P:34}        {t:P:32}        {t:P:35}        {t:P:31}        {t:P:30}
{t:P:27}        {t:P:38}        {t:d:3a}        {t:P:39}        {t:P:2e}        {t:P:2d}        {t:P:37}        {t:P:2c}

{t:P:33}        {t:P:21}        {t:P:36}        {t:P:34}        {t:P:32}        {t:P:35}        {t:P:31}        {t:P:30}
{t:P:27}        {t:P:38}        {t:P:3a}        {t:P:39}        {t:P:2e}        {t:P:2d}        {t:P:37}        {t:P:3b}

{t:P:33}        {t:P:21}        {t:P:36}        {t:P:34}        {t:P:32}        {t:P:35}        {t:P:31}        {t:P:30}
{t:P:3c}        {t:P:38}        {t:P:3a}        {t:P:39}        {t:P:2e}        {t:P:2d}        {t:P:37}        {t:P:3b}

{t:P:33}        {t:d:3d}        {t:P:36}        {t:P:34}        {t:P:32}        {t:P:35}        {t:P:31}        {t:P:30}
{t:P:3c}        {t:P:38}        {t:P:3a}        {t:P:39}        {t:P:2e}        {t:P:2d}        {t:P:37}        {t:P:3b}

{t:P:33}        {t:P:3d}        {t:P:36}        {t:P:34}        {t:P:32}        {t:P:35}        {t:P:31}        {t:P:30}
{t:P:3c}        {t:P:38}        {t:P:3a}        {t:P:39}        {t:P:2e}        {t:P:3e}        {t:P:37}        {t:P:3b}

{t:P:33}        {t:P:3d}        {t:P:36}        {t:P:34}        {t:P:32}        {t:P:35}        {t:P:31}        {t:P:30}
{t:P:3c}        {t:P:38}        {t:P:3a}        {t:P:39}        {t:d:3f}        {t:P:3e}        {t:P:37}        {t:P:3b}

{t:P:33}        {t:P:3d}        {t:P:36}        {t:P:34}        {t:P:32}        {t:P:35}        {t:d:40}        {t:P:30}
{t:P:3c}        {t:P:38}        {t:P:3a}        {t:P:39}        {t:P:3f}        {t:P:3e}        {t:P:37}        {t:P:3b}

{t:d:41}        {t:P:3d}        {t:P:36}        {t:P:34}        {t:P:32}        {t:P:35}        {t:P:40}        {t:P:30}
{t:P:3c}        {t:P:38}        {t:P:3a}        {t:P:39}        {t:P:3f}        {t:P:3e}        {t:P:37}        {t:P:3b}

{t:P:41}        {t:P:3d}        {t:P:36}        {t:P:34}        {t:d:42}        {t:P:35}        {t:P:40}        {t:P:30}
{t:P:3c}        {t:P:38}        {t:P:3a}        {t:P:39}        {t:P:3f}        {t:P:3e}        {t:P:37}        {t:P:3b}

{t:P:41}        {t:P:3d}        {t:P:36}        {t:d:43}        {t:P:42}        {t:P:35}        {t:P:40}        {t:P:30}
{t:P:3c}        {t:P:38}        {t:P:3a}        {t:P:39}        {t:P:3f}        {t:P:3e}        {t:P:37}        {t:P:3b}

{t:P:41}        {t:P:3d}        {t:P:36}        {t:P:43}        {t:P:42}        {t:d:44}        {t:P:40}        {t:P:30}
{t:P:3c}        {t:P:38}        {t:P:3a}        {t:P:39}        {t:P:3f}        {t:P:3e}        {t:P:37}        {t:P:3b}

{t:P:41}        {t:P:3d}        {t:P:36}        {t:P:43}        {t:P:42}        {t:P:44}        {t:P:40}        {t:d:45}
{t:P:3c}        {t:P:38}        {t:P:3a}        {t:P:39}        {t:P:3f}        {t:P:3e}        {t:P:37}        {t:P:3b}

{t:P:41}        {t:P:3d}        {t:d:46}        {t:P:43}        {t:P:42}        {t:P:44}        {t:P:40}        {t:P:45}
{t:P:3c}        {t:P:38}        {t:P:3a}        {t:P:39}        {t:P:3f}        {t:P:3e}        {t:P:37}        {t:P:3b}

{t:P:41}        {t:P:3d}        {t:P:46}        {t:P:43}        {t:P:42}        {t:P:44}        {t:P:40}        {t:P:45}
{t:P:3c}        {t:d:47}        {t:P:3a}        {t:P:39}        {t:P:3f}        {t:P:3e}        {t:P:37}        {t:P:3b}

{t:P:41}        {t:P:3d}        {t:P:46}        {t:P:43}        {t:P:42}        {t:P:44}        {t:P:40}        {t:P:45}
{t:P:3c}        {t:P:47}        {t:P:3a}        {t:P:48}        {t:P:3f}        {t:P:3e}        {t:P:37}        {t:P:3b}

{t:P:41}        {t:P:3d}        {t:P:46}        {t:P:43}        {t:P:42}        {t:P:44}        {t:P:40}        {t:P:45}
{t:P:3c}        {t:P:47}        {t:P:3a}        {t:P:48}        {t:P:3f}        {t:P:3e}        {t:P:37}        {t:P:49}

{t:P:41}        {t:P:3d}        {t:P:46}        {t:P:43}        {t:P:42}        {t:P:44}        {t:P:40}        {t:P:45}
{t:P:3c}        {t:P:47}        {t:P:3a}        {t:P:48}        {t:P:3f}        {t:P:3e}        {t:d:4a}        {t:P:49}

{t:P:41}        {t:P:3d}        {t:P:46}        {t:P:43}        {t:P:42}        {t:P:44}        {t:P:40}        {t:P:45}
{t:d:4b}        {t:P:47}        {t:P:3a}        {t:P:48}        {t:P:3f}        {t:P:3e}        {t:P:4a}        {t:P:49}

{t:P:41}        {t:d:4c}        {t:P:46}        {t:P:43}        {t:P:42}        {t:P:44}        {t:P:40}        {t:P:45}
{t:P:4b}        {t:P:47}        {t:P:3a}        {t:P:48}        {t:P:3f}        {t:P:3e}        {t:P:4a}        {t:P:49}

{t:P:41}        {t:P:4c}        {t:P:46}        {t:P:43}        {t:P:42}        {t:P:44}        {t:P:40}        {t:P:45}
{t:P:4b}        {t:P:47}        {t:d:4d}        {t:P:48}        {t:P:3f}        {t:P:3e}        {t:P:4a}        {t:P:49}

{t:P:41}        {t:P:4c}        {t:P:46}        {t:P:43}        {t:P:42}        {t:P:44}        {t:P:40}        {t:P:45}
{t:P:4b}        {t:P:47}        {t:P:4d}        {t:P:48}        {t:P:3f}        {t:d:4e}        {t:P:4a}        {t:P:49}

{t:P:41}        {t:P:4c}        {t:P:46}        {t:P:43}        {t:P:42}        {t:P:44}        {t:P:40}        {t:P:45}
{t:P:4b}        {t:P:47}        {t:P:4d}        {t:P:48}        {t:P:4f}        {t:P:4e}        {t:P:4a}        {t:P:49}

{t:P:41}        {t:P:4c}        {t:P:46}        {t:i:43}        {t:d:50}        {t:P:44}        {t:P:40}        {t:P:45}
{t:P:4b}        {t:P:47}        {t:P:4d}        {t:P:48}        {t:P:4f}        {t:P:4e}        {t:P:4a}        {t:P:49}

{t:P:41}        {t:P:4c}        {t:P:46}        {t:d:51}        {t:P:50}        {t:P:44}        {t:P:40}        {t:P:45}
{t:P:4b}        {t:P:47}        {t:P:4d}        {t:P:48}        {t:P:4f}        {t:P:4e}        {t:P:4a}        {t:P:49}

{t:P:41}        {t:P:4c}        {t:P:46}        {t:P:51}        {t:P:50}        {t:P:44}        {t:d:52}        {t:P:45}
{t:P:4b}        {t:P:47}        {t:P:4d}        {t:P:48}        {t:P:4f}        {t:P:4e}        {t:P:4a}        {t:P:49}

{t:P:41}        {t:P:4c}        {t:P:46}        {t:P:51}        {t:P:50}        {t:d:53}        {t:P:52}        {t:P:45}
{t:P:4b}        {t:P:47}        {t:P:4d}        {t:P:48}        {t:P:4f}        {t:P:4e}        {t:P:4a}        {t:P:49}

{t:d:54}        {t:P:4c}        {t:P:46}        {t:P:51}        {t:P:50}        {t:P:53}        {t:P:52}        {t:P:45}
{t:P:4b}        {t:P:47}        {t:P:4d}        {t:P:48}        {t:P:4f}        {t:P:4e}        {t:P:4a}        {t:P:49}

{t:P:54}        {t:P:4c}        {t:d:55}        {t:P:51}        {t:P:50}        {t:P:53}        {t:P:52}        {t:P:45}
{t:P:4b}        {t:P:47}        {t:P:4d}        {t:P:48}        {t:P:4f}        {t:P:4e}        {t:P:4a}        {t:P:49}

{t:P:54}        {t:P:4c}        {t:P:55}        {t:P:51}        {t:P:50}        {t:P:53}        {t:P:52}        {t:d:56}
{t:P:4b}        {t:P:47}        {t:P:4d}        {t:P:48}        {t:P:4f}        {t:P:4e}        {t:P:4a}        {t:P:49}

{t:P:54}        {t:P:4c}        {t:P:55}        {t:P:51}        {t:P:50}        {t:P:53}        {t:P:52}        {t:P:56}
{t:P:4b}        {t:P:47}        {t:P:4d}        {t:P:48}        {t:P:4f}        {t:P:4e}        {t:P:4a}        {t:P:57}

{t:P:54}        {t:P:4c}        {t:P:55}        {t:P:51}        {t:P:50}        {t:P:53}        {t:P:52}        {t:P:56}
{t:P:4b}        {t:P:58}        {t:P:4d}        {t:P:48}        {t:P:4f}        {t:P:4e}        {t:P:4a}        {t:P:57}

{t:P:54}        {t:d:59}        {t:P:55}        {t:P:51}        {t:P:50}        {t:P:53}        {t:P:52}        {t:P:56}
{t:P:4b}        {t:P:58}        {t:P:4d}        {t:P:48}        {t:P:4f}        {t:P:4e}        {t:P:4a}        {t:P:57}

{t:P:54}        {t:P:59}        {t:P:55}        {t:P:51}        {t:P:50}        {t:P:53}        {t:P:52}        {t:P:56}
{t:P:4b}        {t:P:58}        {t:P:4d}        {t:P:48}        {t:P:4f}        {t:P:4e}        {t:d:5a}        {t:P:57}

{t:P:54}        {t:P:59}        {t:P:55}        {t:P:51}        {t:P:50}        {t:P:53}        {t:P:52}        {t:P:56}
{t:P:4b}        {t:P:58}        {t:P:4d}        {t:P:5b}        {t:P:4f}        {t:P:4e}        {t:P:5a}        {t:P:57}

{t:P:54}        {t:P:59}        {t:P:55}        {t:P:51}        {t:P:50}        {t:P:53}        {t:P:52}        {t:P:56}
{t:P:4b}        {t:P:58}        {t:P:4d}        {t:P:5b}        {t:P:4f}        {t:d:5c}        {t:P:5a}        {t:P:57}

{t:P:54}        {t:P:59}        {t:P:55}        {t:P:51}        {t:P:50}        {t:P:53}        {t:P:52}        {t:P:56}
{t:P:4b}        {t:P:58}        {t:d:5d}        {t:P:5b}        {t:P:4f}        {t:P:5c}        {t:P:5a}        {t:P:57}

{t:P:54}        {t:P:59}        {t:P:55}        {t:P:51}        {t:P:50}        {t:P:53}        {t:P:52}        {t:P:56}
{t:d:5e}        {t:P:58}        {t:P:5d}        {t:P:5b}        {t:P:4f}        {t:P:5c}        {t:P:5a}        {t:P:57}

{t:P:54}        {t:P:59}        {t:P:55}        {t:P:51}        {t:P:50}        {t:P:53}        {t:P:52}        {t:P:56}
{t:P:5e}        {t:P:58}        {t:P:5d}        {t:P:5b}        {t:d:5f}        {t:P:5c}        {t:P:5a}        {t:P:57}

{t:P:54}        {t:P:59}        {t:d:60}        {t:P:51}        {t:P:50}        {t:P:53}        {t:P:52}        {t:P:56}
{t:P:5e}        {t:P:58}        {t:P:5d}        {t:P:5b}        {t:P:5f}        {t:P:5c}        {t:P:5a}        {t:P:57}

{t:P:54}        {t:P:59}        {t:P:60}        {t:P:51}        {t:P:50}        {t:P:61}        {t:P:52}        {t:P:56}
{t:P:5e}        {t:P:58}        {t:P:5d}        {t:P:5b}        {t:P:5f}        {t:P:5c}        {t:P:5a}        {t:P:57}

{t:P:54}        {t:P:59}        {t:P:60}        {t:d:62}        {t:P:50}        {t:P:61}        {t:P:52}        {t:P:56}
{t:P:5e}        {t:P:58}        {t:P:5d}        {t:P:5b}        {t:P:5f}        {t:P:5c}        {t:P:5a}        {t:P:57}

{t:P:54}        {t:P:59}        {t:P:60}        {t:P:62}        {t:P:63}        {t:P:61}        {t:P:52}        {t:P:56}
{t:P:5e}        {t:P:58}        {t:P:5d}        {t:P:5b}        {t:P:5f}        {t:P:5c}        {t:P:5a}        {t:P:57}

{t:P:54}        {t:P:59}        {t:P:60}        {t:P:62}        {t:P:63}        {t:P:61}        {t:d:64}        {t:P:56}
{t:P:5e}        {t:P:58}        {t:P:5d}        {t:P:5b}        {t:P:5f}        {t:P:5c}        {t:P:5a}        {t:P:57}

{t:P:54}        {t:P:59}        {t:P:60}        {t:P:62}        {t:P:63}        {t:P:61}        {t:P:64}        {t:P:56}
{t:P:5e}        {t:P:58}        {t:P:5d}        {t:P:5b}        {t:P:5f}        {t:P:5c}        {t:P:5a}        {t:d:65}

{t:d:66}        {t:P:59}        {t:P:60}        {t:P:62}        {t:P:63}        {t:P:61}        {t:P:64}        {t:P:56}
{t:P:5e}        {t:P:58}        {t:P:5d}        {t:P:5b}        {t:P:5f}        {t:P:5c}        {t:P:5a}        {t:P:65}

{t:P:66}        {t:d:67}        {t:P:60}        {t:P:62}        {t:P:63}        {t:P:61}        {t:P:64}        {t:P:56}
{t:P:5e}        {t:P:58}        {t:P:5d}        {t:P:5b}        {t:P:5f}        {t:P:5c}        {t:P:5a}        {t:P:65}

{t:P:66}        {t:P:67}        {t:P:60}        {t:P:62}        {t:P:63}        {t:P:61}        {t:P:64}        {t:d:68}
{t:P:5e}        {t:P:58}        {t:P:5d}        {t:P:5b}        {t:P:5f}        {t:P:5c}        {t:P:5a}        {t:P:65}

{t:P:66}        {t:P:67}        {t:P:60}        {t:P:62}        {t:P:63}        {t:P:61}        {t:P:64}        {t:P:68}
{t:P:5e}        {t:P:58}        {t:P:5d}        {t:P:5b}        {t:P:5f}        {t:P:5c}        {t:d:69}        {t:P:65}

{t:P:66}        {t:P:67}        {t:P:60}        {t:P:62}        {t:P:63}        {t:P:61}        {t:P:64}        {t:P:68}
{t:P:5e}        {t:d:6a}        {t:P:5d}        {t:P:5b}        {t:P:5f}        {t:P:5c}        {t:P:69}        {t:P:65}

{t:P:66}        {t:P:67}        {t:P:60}        {t:P:62}        {t:P:63}        {t:P:61}        {t:P:64}        {t:P:68}
{t:P:5e}        {t:P:6a}        {t:P:5d}        {t:P:5b}        {t:P:5f}        {t:d:6b}        {t:P:69}        {t:P:65}

{t:P:66}        {t:P:67}        {t:P:60}        {t:P:62}        {t:P:63}        {t:P:61}        {t:P:64}        {t:P:68}
{t:P:5e}        {t:P:6a}        {t:d:6c}        {t:P:5b}        {t:P:5f}        {t:P:6b}        {t:P:69}        {t:P:65}

{t:P:66}        {t:P:67}        {t:P:60}        {t:P:62}        {t:P:63}        {t:P:61}        {t:P:64}        {t:P:68}
{t:d:6d}        {t:P:6a}        {t:P:6c}        {t:P:5b}        {t:P:5f}        {t:P:6b}        {t:P:69}        {t:P:65}

{t:P:66}        {t:P:67}        {t:P:60}        {t:P:62}        {t:P:63}        {t:P:61}        {t:P:64}        {t:P:68}
{t:P:6d}        {t:P:6a}        {t:P:6c}        {t:d:6e}        {t:P:5f}        {t:P:6b}        {t:P:69}        {t:P:65}

{t:P:66}        {t:P:67}        {t:P:6f}        {t:P:62}        {t:P:63}        {t:P:61}        {t:P:64}        {t:P:68}
{t:P:6d}        {t:P:6a}        {t:P:6c}        {t:P:6e}        {t:P:5f}        {t:P:6b}        {t:P:69}        {t:P:65}

{t:P:66}        {t:P:67}        {t:P:6f}        {t:P:62}        {t:P:63}        {t:P:61}        {t:P:64}        {t:P:68}
{t:P:6d}        {t:P:6a}        {t:P:6c}        {t:P:6e}        {t:d:70}        {t:P:6b}        {t:P:69}        {t:P:65}
{t:P:54}        {t:P:59}        {t:P:60}        {t:P:62}        {t:P:63}        {t:P:61}        {t:d:64}        {t:P:56}
{t:P:5e}        {t:P:58}        {t:P:5d}        {t:P:5b}        {t:P:5f}        {t:P:5c}        {t:P:5a}        {t:P:57}

{t:P:54}        {t:P:59}        {t:P:60}        {t:P:62}        {t:P:63}        {t:P:61}        {t:P:64}        {t:P:56}
{t:P:5e}        {t:P:58}        {t:P:5d}        {t:P:5b}        {t:P:5f}        {t:P:5c}        {t:P:5a}        {t:d:65}

{t:d:66}        {t:P:59}        {t:P:60}        {t:P:62}        {t:P:63}        {t:P:61}        {t:P:64}        {t:P:56}
{t:P:5e}        {t:P:58}        {t:P:5d}        {t:P:5b}        {t:P:5f}        {t:P:5c}        {t:P:5a}        {t:P:65}

{t:P:66}        {t:d:67}        {t:P:60}        {t:P:62}        {t:P:63}        {t:P:61}        {t:P:64}        {t:P:56}
{t:P:5e}        {t:P:58}        {t:P:5d}        {t:P:5b}        {t:P:5f}        {t:P:5c}        {t:P:5a}        {t:P:65}

{t:P:66}        {t:P:67}        {t:P:60}        {t:P:62}        {t:P:63}        {t:P:61}        {t:P:64}        {t:d:68}
{t:P:5e}        {t:P:58}        {t:P:5d}        {t:P:5b}        {t:P:5f}        {t:P:5c}        {t:P:5a}        {t:P:65}

{t:P:66}        {t:P:67}        {t:P:60}        {t:P:62}        {t:P:63}        {t:P:61}        {t:P:64}        {t:P:68}
{t:P:5e}        {t:P:58}        {t:P:5d}        {t:P:5b}        {t:P:5f}        {t:P:5c}        {t:d:69}        {t:P:65}

{t:P:66}        {t:P:67}        {t:P:60}        {t:P:62}        {t:P:63}        {t:P:61}        {t:P:64}        {t:P:68}
{t:P:5e}        {t:d:6a}        {t:P:5d}        {t:P:5b}        {t:P:5f}        {t:P:5c}        {t:P:69}        {t:P:65}

{t:P:66}        {t:P:67}        {t:P:60}        {t:P:62}        {t:P:63}        {t:P:61}        {t:P:64}        {t:P:68}
{t:P:5e}        {t:P:6a}        {t:P:5d}        {t:P:5b}        {t:P:5f}        {t:d:6b}        {t:P:69}        {t:P:65}

{t:P:66}        {t:P:67}        {t:P:60}        {t:P:62}        {t:P:63}        {t:P:61}        {t:P:64}        {t:P:68}
{t:P:5e}        {t:P:6a}        {t:d:6c}        {t:P:5b}        {t:P:5f}        {t:P:6b}        {t:P:69}        {t:P:65}

{t:P:66}        {t:P:67}        {t:P:60}        {t:P:62}        {t:P:63}        {t:P:61}        {t:P:64}        {t:P:68}
{t:d:6d}        {t:P:6a}        {t:P:6c}        {t:P:5b}        {t:P:5f}        {t:P:6b}        {t:P:69}        {t:P:65}

{t:P:66}        {t:P:67}        {t:P:60}        {t:P:62}        {t:P:63}        {t:P:61}        {t:P:64}        {t:P:68}
{t:P:6d}        {t:P:6a}        {t:P:6c}        {t:d:6e}        {t:P:5f}        {t:P:6b}        {t:P:69}        {t:P:65}

{t:P:66}        {t:P:67}        {t:P:6f}        {t:P:62}        {t:P:63}        {t:P:61}        {t:P:64}        {t:P:68}
{t:P:6d}        {t:P:6a}        {t:P:6c}        {t:P:6e}        {t:P:5f}        {t:P:6b}        {t:P:69}        {t:P:65}

{t:P:66}        {t:P:67}        {t:P:6f}        {t:P:62}        {t:P:63}        {t:P:61}        {t:P:64}        {t:P:68}
{t:P:6d}        {t:P:6a}        {t:P:6c}        {t:P:6e}        {t:d:70}        {t:P:6b}        {t:P:69}        {t:P:65}

{t:P:66}        {t:P:67}        {t:P:6f}        {t:P:62}        {t:P:63}        {t:d:71}        {t:P:64}        {t:P:68}
{t:P:6d}        {t:P:6a}        {t:P:6c}        {t:P:6e}        {t:P:70}        {t:P:6b}        {t:P:69}        {t:P:65}

{t:P:66}        {t:P:67}        {t:P:6f}        {t:d:72}        {t:P:63}        {t:P:71}        {t:P:64}        {t:P:68}
{t:P:6d}        {t:P:6a}        {t:P:6c}        {t:P:6e}        {t:P:70}        {t:P:6b}        {t:P:69}        {t:P:65}

{t:P:66}        {t:P:67}        {t:P:6f}        {t:P:72}        {t:P:63}        {t:P:71}        {t:d:73}        {t:P:68}
{t:P:6d}        {t:P:6a}        {t:P:6c}        {t:P:6e}        {t:P:70}        {t:P:6b}        {t:P:69}        {t:P:65}

{t:P:66}        {t:P:67}        {t:P:6f}        {t:P:72}        {t:P:74}        {t:P:71}        {t:P:73}        {t:P:68}
{t:P:6d}        {t:P:6a}        {t:P:6c}        {t:P:6e}        {t:P:70}        {t:P:6b}        {t:P:69}        {t:P:65}

{t:P:66}        {t:P:67}        {t:P:6f}        {t:P:72}        {t:P:74}        {t:P:71}        {t:P:73}        {t:P:68}
{t:P:6d}        {t:P:6a}        {t:P:6c}        {t:P:6e}        {t:P:70}        {t:P:6b}        {t:P:69}        {t:d:75}

{t:d:76}        {t:P:67}        {t:P:6f}        {t:P:72}        {t:P:74}        {t:P:71}        {t:P:73}        {t:P:68}
{t:P:6d}        {t:P:6a}        {t:P:6c}        {t:P:6e}        {t:P:70}        {t:P:6b}        {t:P:69}        {t:P:75}

{t:P:76}        {t:d:77}        {t:P:6f}        {t:P:72}        {t:P:74}        {t:P:71}        {t:P:73}        {t:P:68}
{t:P:6d}        {t:P:6a}        {t:P:6c}        {t:P:6e}        {t:P:70}        {t:P:6b}        {t:P:69}        {t:P:75}

{t:P:76}        {t:P:77}        {t:P:6f}        {t:P:72}        {t:P:74}        {t:P:71}        {t:P:73}        {t:P:78}
{t:P:6d}        {t:P:6a}        {t:P:6c}        {t:P:6e}        {t:P:70}        {t:P:6b}        {t:P:69}        {t:P:75}

{t:P:76}        {t:P:77}        {t:P:6f}        {t:P:72}        {t:P:74}        {t:P:71}        {t:P:73}        {t:P:78}
{t:P:6d}        {t:P:6a}        {t:P:6c}        {t:P:6e}        {t:P:70}        {t:P:6b}        {t:P:79}        {t:P:75}
^T
load: 0.31  cmd: viamillipede 15810 [nanslp] 69.73r 0.32u 0.70s 0% 37952k

244(MiBytes)/69(s)=  3.6690(MiBps)
{t:P:76}        {t:P:77}        {t:P:6f}        {t:P:72}        {t:P:74}        {t:P:71}        {t:P:73}        {t:P:78}
{t:P:6d}        {t:P:6a}        {t:P:6c}        {t:P:6e}        {t:P:70}        {t:P:6b}        {t:P:79}        {t:P:75}

^C

@dch
Copy link
Author

dch commented Jan 20, 2021

this time with a few ^T sprinkled at both ends:

# verbose 6 tx
{t:P:81}        {t:P:78}        {t:P:79}        {t:P:7f}        {t:P:7a}        {t:P:86}        {t:P:7c}        {t:P:80}
{t:P:85}        {t:P:83}        {t:P:87}        {t:P:7e}        {t:P:84}        {t:P:82}        {t:P:7d}        {t:P:7b}

{t:P:81}        {t:P:78}        {t:d:88}        {t:P:7f}        {t:P:7a}        {t:P:86}        {t:P:7c}        {t:P:80}
{t:P:85}        {t:P:83}        {t:P:87}        {t:P:7e}        {t:P:84}        {t:P:82}        {t:P:7d}        {t:P:7b}

load: 0.30  cmd: viamillipede 15672 [nanslp] 41.11r 0.16u 0.72s 0% 37960k

274(MiBytes)/42(s)=  6.8228(MiBps)
{t:P:81}        {t:P:78}        {t:P:88}        {t:P:7f}        {t:P:7a}        {t:P:86}        {t:P:7c}        {t:P:80}
{t:P:85}        {t:P:83}        {t:P:87}        {t:P:7e}        {t:P:84}        {t:P:82}        {t:P:7d}        {t:P:7b}

load: 0.36  cmd: zfs 39937 [pipdwt] 44.89r 0.15u 0.70s 0% 18832k

274(MiBytes)/44(s)=  6.3986(MiBps)
{t:P:81}        {t:P:78}        {t:P:88}        {t:P:7f}        {t:P:7a}        {t:P:86}        {t:P:7c}        {t:P:80}
{t:P:85}        {t:P:83}        {t:P:87}        {t:P:7e}        {t:P:84}        {t:P:82}        {t:P:7d}        {t:P:7b}

load: 0.36  cmd: zfs 39937 [pipdwt] 47.11r 0.15u 0.70s 0% 18832k

274(MiBytes)/48(s)=  5.9709(MiBps)
{t:P:81}        {t:P:78}        {t:P:88}        {t:P:7f}        {t:P:7a}        {t:P:86}        {t:P:7c}        {t:P:80}
{t:P:85}        {t:P:83}        {t:P:87}        {t:P:7e}        {t:P:84}        {t:P:82}        {t:P:7d}        {t:P:7b}

^C⏎                                                                                                                                 

# verbose 6 rx
being a server at port 7000 

 rxw:00 accepted fd:5 
rxw:01 accepted fd:6 
rxw:02 accepted fd:7 
rxw:03 accepted fd:8 
rxw:04 accepted fd:9 
rxw:05 accepted fd:10 
rxw:06 accepted fd:11 
rxw:07 accepted fd:12 
rxw:08 accepted fd:13 
rxw:09 accepted fd:14 
rxw:10 accepted fd:15 
rxw:11 accepted fd:16 
rxw:12 accepted fd:17 
rxw:13 accepted fd:18 
rxw:14 accepted fd:19 
rxw:15 accepted fd:20 
load: 0.32  cmd: viamillipede 67886 [nanslp] 6.14r 0.06u 0.25s 0% 27012k
mi_switch+0xc1 sleepq_catch_signals+0x2e6 sleepq_timedwait_sig+0x12 _sleep+0x199 kern_clock_nanosleep+0x1e1 sys_nanosleep+0x3b amd64_syscall+0x10c fast_syscall_common+0xf8 
rxconf: workers 16, sequencerleg: 0 done: 0
{r:01:r:1:o}	{r:02:r:2:o}	{r:03:r:3:o}	{r:04:r:4:o}	{r:05:r:5:o}	{r:06:r:6:o}	{r:07:r:7:o}	
{r:08:r:8:o}	{r:09:r:9:o}	{r:10:r:a:o}	{r:11:r:b:o}	{r:12:r:c:o}	{r:13:r:d:o}	{r:14:r:e:o}	{r:15:r:f:o}	

{t::0}	{t::0}	{t::0}	{t::0}	{t::0}	{t::0}	{t::0}	{t::0}	
{t::0}	{t::0}	{t::0}	{t::0}	{t::0}	{t::0}	{t::0}	{t::0}	

load: 0.29  cmd: viamillipede 67886 [nanslp] 7.94r 0.08u 0.38s 0% 36932k
mi_switch+0xc1 sleepq_catch_signals+0x2e6 sleepq_timedwait_sig+0x12 _sleep+0x199 kern_clock_nanosleep+0x1e1 sys_nanosleep+0x3b amd64_syscall+0x10c fast_syscall_common+0xf8 
rxconf: workers 16, sequencerleg: 10 done: 0
{r:00:r:10:$}	{r:01:r:11:o}	{r:02:r:12:o}	{r:03:r:13:o}	{r:04:r:15:o}	{r:05:r:14:o}	{r:06:r:16:o}	{r:07:r:17:o}	
{r:08:r:18:o}	{r:09:r:19:o}	{r:10:r:1a:o}	{r:11:r:1b:o}	{r:12:r:1d:o}	{r:13:r:1c:o}	{r:14:r:1e:o}	{r:15:r:1f:o}	

load: 0.29  cmd: viamillipede 67886 [nanslp] 9.79r 0.09u 0.55s 0% 36932k
mi_switch+0xc1 sleepq_catch_signals+0x2e6 sleepq_timedwait_sig+0x12 _sleep+0x199 kern_clock_nanosleep+0x1e1 sys_nanosleep+0x3b amd64_syscall+0x10c fast_syscall_common+0xf8 
rxconf: workers 16, sequencerleg: 10 done: 0
{r:00:r:10:$}	{r:01:r:11:o}	{r:02:r:12:o}	{r:03:r:13:o}	{r:04:r:15:o}	{r:05:r:14:o}	{r:06:r:16:o}	{r:07:r:17:o}	
{r:08:r:18:o}	{r:09:r:19:o}	{r:10:r:1a:o}	{r:11:r:1b:o}	{r:12:r:1d:o}	{r:13:r:1c:o}	{r:14:r:1e:o}	{r:15:r:1f:o}	

load: 0.27  cmd: viamillipede 67886 [nanslp] 11.55r 0.11u 0.67s 0% 36932k
mi_switch+0xc1 sleepq_catch_signals+0x2e6 sleepq_timedwait_sig+0x12 _sleep+0x199 kern_clock_nanosleep+0x1e1 sys_nanosleep+0x3b amd64_syscall+0x10c fast_syscall_common+0xf8 
rxconf: workers 16, sequencerleg: 10 done: 0
{r:00:r:10:$}	{r:01:r:11:o}	{r:02:r:12:o}	{r:03:r:13:o}	{r:04:r:15:o}	{r:05:r:14:o}	{r:06:r:16:o}	{r:07:r:17:o}	
{r:08:r:18:o}	{r:09:r:19:o}	{r:10:r:1a:o}	{r:11:r:1b:o}	{r:12:r:1d:o}	{r:13:r:1c:o}	{r:14:r:1e:o}	{r:15:r:1f:o}	

load: 0.27  cmd: viamillipede 67886 [running] 16.60r 0.18u 0.91s 0% 36932k
mi_switch+0xc1 sleepq_catch_signals+0x2e6 sleepq_timedwait_sig+0x12 _sleep+0x199 kern_clock_nanosleep+0x1e1 sys_nanosleep+0x3b amd64_syscall+0x10c fast_syscall_common+0xf8 
rxconf: workers 16, sequencerleg: 21 done: 0
{r:00:r:35:o}	{r:01:r:21:$}	{r:02:r:22:o}	{r:03:R:23:o}	{r:04:r:25:o}	{r:05:R:24:o}	{r:06:R:27:o}	{r:07:R:26:o}	
{r:08:R:28:o}	{r:09:R:2a:o}	{r:10:R:29:o}	{r:11:R:2b:o}	{r:12:R:2d:o}	{r:13:R:2c:o}	{r:14:R:2e:o}	{r:15:R:2f:o}	

load: 0.31  cmd: viamillipede 67886 [nanslp] 23.12r 0.24u 1.15s 0% 36932k
mi_switch+0xc1 sleepq_catch_signals+0x2e6 sleepq_timedwait_sig+0x12 _sleep+0x199 kern_clock_nanosleep+0x1e1 sys_nanosleep+0x3b amd64_syscall+0x10c fast_syscall_common+0xf8 
rxconf: workers 16, sequencerleg: 3d done: 0
{r:00:r:47:o}	{r:01:r:3f:o}	{r:02:r:3d:$}	{r:03:r:43:o}	{r:04:r:3e:o}	{r:05:r:40:o}	{r:06:r:42:o}	{r:07:r:41:o}	
{r:08:R:44:o}	{r:09:r:49:o}	{r:10:r:48:o}	{r:11:r:4c:o}	{r:12:r:4a:o}	{r:13:r:4b:o}	{r:14:r:46:o}	{r:15:r:45:o}	

load: 0.34  cmd: viamillipede 67886 [nanslp] 32.25r 0.37u 1.50s 0% 36932k
mi_switch+0xc1 sleepq_catch_signals+0x2e6 sleepq_timedwait_sig+0x12 _sleep+0x199 kern_clock_nanosleep+0x1e1 sys_nanosleep+0x3b amd64_syscall+0x10c fast_syscall_common+0xf8 
rxconf: workers 16, sequencerleg: 56 done: 0
{r:00:r:5e:o}	{r:01:R:57:o}	{r:02:r:67:o}	{r:03:r:59:o}	{r:04:r:5d:o}	{r:05:r:65:o}	{r:06:r:60:o}	{r:07:r:5f:o}	
{r:08:R:5c:o}	{r:09:R:58:o}	{r:10:r:63:o}	{r:11:r:5b:o}	{r:12:r:64:o}	{r:13:R:5a:o}	{r:14:r:61:o}	{r:15:r:56:$}	

Assertion failed: (pkt.preamble == preamble_cannon_ul && "preamble check"), function rxworker, file rx.c, line 63.

________________________________________________________
Executed in   39.26 secs   fish           external 
   usr time    0.47 secs  782.00 micros    0.47 secs 
   sys time    2.15 secs    0.00 micros    2.15 secs 

fish: “time viamillipede verbose 6 rx…” terminated by signal SIGABRT (Abort)

@agokhale
Copy link
Owner

I'll take the pcap now :) , but would you try limiting 'threads 14' on the tx side ?

@dch
Copy link
Author

dch commented Jan 20, 2021

this only seems to happen with zfs send, if I use anything else, we're good. how curious. I'll need to wait til tomorrow for a pcap, how can I send it privately?

@agokhale
Copy link
Owner

I can reproduce this locally; no pcap needed. Stay tuned.

@dch
Copy link
Author

dch commented Feb 22, 2023

any progress on this? would be nice to bump the port off 0.89 too

@agokhale
Copy link
Owner

agokhale commented Mar 9, 2023

can you try the current head and see if the buffer changes alter the behaviour. I'd love to crack this, but at the moment it's important that folks send over their bugs.

@dch
Copy link
Author

dch commented Mar 11, 2023

still flakey - this is transferring a bit under 100GiB between 2 boxes within the same DC.
They are both lightly loaded although the first one has sucky disk i/o.

I initially tried via zfs send .. | viamillipede ... but switched to a plain file after repeated fails. This one is from the file:

# viamillipede verbose 6 tx 193.122.58.101 2200 < i09.zfs
...
broken pipe

# viamillipede verbose 6  rx 2200 > i09.zfs

rxw:10 accepted fd:16
rxw:09 accepted fd:17
rxw:02 accepted fd:18
rxw:15 accepted fd:19
{t::0}  {t::0}  {t::0}  {t::0}  {t::0}  {t::0}  {t::0}  {t::0}
{t::0}  {t::0}  {t::0}  {t::0}  {t::0}  {t::0}  {t::0}  {t::0}

readlen:0x24 expected 0x28rxconf: workers 16, sequencerleg: 41dd done: 0
{r:00:R:41ec:o} {r:01:R:41e7:o} {r:02:a:41dc:o} {r:03:R:41de:o} {r:04:P:41dd:$}      {r:05:R:41e8:o} {r:06:R:41ee:o} {r:07:R:41e2:o}
{r:08:R:41e4:o} {r:09:R:41e3:o} {r:10:R:41e5:o} {r:11:R:41f1:o} {r:12:R:41e0:o}      {r:13:R:41e6:o} {r:14:r:41ed:o} {r:15:r:41df:o}

preamble check failure got 5a5a5a5a915eb8dd != a5a5a5a55a5a5a5arxconf: workers 16, sequencerleg: 41f4 done: 0
{r:00:R:41f7:o} {r:01:R:41fb:o} {r:02:a:41f3:o} {r:03:R:41f9:o} {r:04:R:4206:o}      {r:05:R:41fd:o} {r:06:r:4203:o} {r:07:R:41fa:o}
{r:08:P:41f4:$} {r:09:r:4214:o} {r:10:R:41fe:o} {r:11:r:41f6:o} {r:12:R:41f8:o}      {r:13:R:41fc:o} {r:14:R:41f5:o} {r:15:R:4216:o}

Assertion failed: (pkt.size <= kfootsize), function rxworker, file /var/tmp/viamillipede/rx.c, line 72.
Abort trap

@dch
Copy link
Author

dch commented Mar 11, 2023

and another one, much earlier on this time

rxw:15 accepted fd:19
{t::0}  {t::0}  {t::0}  {t::0}  {t::0}  {t::0}  {t::0}  {t::0}
{t::0}  {t::0}  {t::0}  {t::0}  {t::0}  {t::0}  {t::0}  {t::0}

readlen:0x1c expected 0x28rxconf: workers 16, sequencerleg: 4efe done: 0
{r:00:R:4f13:o} {r:01:R:4f00:o} {r:02:R:4f10:o} {r:03:R:4f05:o} {r:04:R:4f03:o} {r:05:r:4f0e:o} {r:06:R:4eff:o}  {r:07:R:4f06:o}
{r:08:R:4f02:o} {r:09:R:4f04:o} {r:10:a:4efd:o} {r:11:R:4f12:o} {r:12:P:4efe:$} {r:13:R:4f07:o} {r:14:R:4f01:o}  {r:15:R:4f0b:o}

preamble check failure got 2450f0af609f3 != a5a5a5a55a5a5a5arxconf: workers 16, sequencerleg: 4f18 done: 0
{r:00:r:4f1c:o} {r:01:R:4f29:o} {r:02:r:4f20:o} {r:03:R:4f1b:o} {r:04:R:4f1f:o} {r:05:r:4f24:o} {r:06:r:4f18:$}  {r:07:r:4f1e:o}
{r:08:r:4f23:o} {r:09:r:4f21:o} {r:10:a:4f17:o} {r:11:R:4f22:o} {r:12:R:4f1a:o} {r:13:R:4f1d:o} {r:14:R:4f19:o}  {r:15:r:4f36:o}

Assertion failed: (pkt.size <= kfootsize), function rxworker, file /var/tmp/viamillipede/rx.c, line 72.
Abort trap

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dch @agokhale