Sending data in MB takes long time

[ManojKumarChauhan]

I am using libwebsockets to send data using WSS connection. I noticed that the library sends max 60 mb of data per minute.
To send data I followed following steps:

1. Create the connection
2. call lws_callback_on_writable
3. call lws_service in a loop until it sends all the data
4. Writing data in a callback LWS_CALLBACK_CLIENT_WRITEABLE using lws_write

Please suggest what wrong I am doing. Is there any way to increase the speed?

[lws-team]

I noticed that the library sends max 60 mb of data per minute.

You noticed that the library, plus your code / config, on your platform, with your connection, and your specific route between the two peers, acts a particular way. Depending on the details of all those things, sending 1MB/sec might be perfectly reasonable.

I would look first at how large are the individual packets coming out, and what size are you asking lws to send each time? Where are you sending it? Does it act differently if the peer is on 127.0.0.1 / lo and no actual internet involved?

[ManojKumarChauhan]

I am reading data from a file & sending 512 kb each time. Memory for 512 kb buffer is allocated before reading the file. It's on local network from machine 1 to machine 2. I am using libwebsocket on windows & sending on linux server.
No memory allocation while sending data besides IO operation to read 512 kb each time from a file.
One application using ssl library sends data at very high speed. I will try to use 1 mb of buffer.
Any other suggestion?

[lws-team]

Yeah don't try to send large amounts like 512KB at one time, it cannot actually succeed to do it unless the tcp window has space for it. What it does instead is accept what you asked for, and buffer again what couldn't actually be sent in the tcp window by doing it in the background.

So you should think about sending eg 4KB or at least no more than 16KB, which is the size of the tls buffers for the connection. You should generate the next 16KB to be sent in the writeable callback and ask for another writeable callback while still more to send. How long it takes to get the next writeable callback depends on what you're sending and the tcp window situation as ACKs arrive.

Check the sizes of packets actually going out on the wire, eg, with wireshark.

[ManojKumarChauhan]

There is no major impact after using 1 mb of buffer but when I used buffersize 16 kb (i.e. sending 16 kb) transfer speed is reduced by half. Attached screen shot (from wireshark) gives details of windows size(default tcp) & data/packet size.

sample code to transfer data for each 16 kb

while(1) {
read 16 kb from file
transfersize = 16 kb
lws_callback_on_writable(wsi)

while (rv >= 0 && transfersize > 0 ) {
rv = lws_service(context, 0);
}
}

code in callback LWS_CALLBACK_CLIENT_WRITEABLE
int pos = 0, sent = 0;
while (transfersize > 0) {
sent += lws_write(wsi, (unsigned char*)buf + LWS_PRE+pos, 16kb, LWS_WRITE_TEXT);
pos += sent;
transfersize -= sent;
}

[ManojKumarChauhan]

I think I'm not able to upload the image so writing some details here

Frame 114074: 1272 bytes on wire (10176 bits), 1272 bytes captured (10176 bits) on interface
Internet Protocol Version 4
[TCP Segment Len: 1218]
Flags: 0x018 (PSH, ACK)
Window: 63007
[Calculated window size: 63007]
[Window size scaling factor: -1 (unknown)]
Checksum: 0x69de [unverified]
[Checksum Status: Unverified]
Urgent Pointer: 0
[Timestamps]
[SEQ/ACK analysis]
TCP payload (1218 bytes)
TCP segment data (1218 bytes)
[2 Reassembled TCP Segments (4138 bytes): #114072(2920), #114074(1218)]
[Frame: 114072, payload: 0-2919 (2920 bytes)]
[Frame: 114074, payload: 2920-4137 (1218 bytes)]
[Segment count: 2]
[Reassembled TCP length: 4138]
[Reassembled TCP Data […]:

[lws-team]

... that's not how event loops work... please follow the examples provided instead of making up your own.

You need to ask for lws_callback_on_writable() every time you want to write something. And in the WRITEABLE callback, it is an indication that you have the go-ahead to write something. After writing it, 4K, 16K or whatever, if something else to write then you must ask for lws_callback_on_writable() again and exit the callback. When it's possible to write on the connection again, you will get the next WRITEABLE callback.

POSIX indicates a POLLOUT event on the connection when you can write "something". It doesn't indicate how much; it depends on how many packets where ACKed, clearing space for you to send more. If you write more than the kernel will accept, lws has to buffer the remainder and send it in the background, If you take this wrong approach where you imagine you can loop "sending" everything in one callback, you are just buffering all the data locally and are made to wait until it is actually issued in the background.

Network programming does not work sitting there in a loop spamming things that can't be sent. How it can work is you will fill up the tcp window and have to wait to send more, the gap between asking for the writeable callback and getting it is regulating your production of outgoing data vs the connection's ability to absorb it.

[ManojKumarChauhan]

I checked examples & follow the same steps. Changed the callback code

int pos = 0, sent = 0;
while (transfersize > 0) {
sent += lws_write(wsi, (unsigned char*)buf + LWS_PRE+pos, 16kb, LWS_WRITE_TEXT);
pos += sent;
transfersize -= sent;
}

lws_callback_on_writable(wsi);

But still there is same issue.
Could you please help me to know which version of web socket I should use.

[lws-team]

Nope... show me the example where we sit in a loop doing more than one lws_write() in a writeable callback?

The examples do not work like that because I know that if we get a POLLOUT event, if we are bulk-writing, it usually means we have space in the tcp window only for a handful of packets (this space created by hearing about ACKs received from the peer). It's stupid then to sit in a loop filling up heap with buffered copies we cannot send until later. We should emit a small amount, ask for a callback when more writeable, and then go back to the event loop. In particular, each WRITEABLE should only produce a single lws_write().

[ManojKumarChauhan]

There is no such case when it was executing more than one time inside the loop in callback. It was just a safe check.
I have removed the loop. Trying to send 4 kb data but no success.
Let me try the latest version 4.3.3

[ManojKumarChauhan]

The problem didn't solve after using the latest version 4.3.3. It's uploading 350 mb of data in 5 mins.
Any suggest where to debug in websocket code. I will try to know what is actually taking time.