Crash on multiple async reqs #161
Comments
|
have you found any solution @dr3adx ? |
|
@dr3adx @umerm64 AsyncTCP is a wonderful library riddled with bugs, there has been lots of Pull request to fix these, they are never merged, it's just too sad for something as good as this. |
|
@dr3adx I used to have trouble with the ESP32 getting Panics and rebooting when receiving multiple requests, but after applying the pull requests in my fork it is much more stable now. Did you remeber to increase the CONFIG_ASYNC_TCP_QUEUE_SIZE to something bigger than the default 32? I'm running -DCONFIG_ASYNC_TCP_QUEUE_SIZE=512 |
Hello bro, yes i've tried that, but my issue is when sending multiple requests from multiple AsyncClients. Basically web scraping. I would also get some crash along the lines: tcp_rcv_wnd < 0xFFFF What crash specifically did you get before? My case can be easily reproduced by making a system which should send API req to 10 different websites, each holding a unique cookie and to a different URL so its a session. So initializing 10 AsyncClient objects is where the problem happens I think, because with low amount of async reqs, e.g 3 the issue doesnt happen. But then again, even 10 is nothing, 100+ async reqs in a given time can be easily handled in nodejs for comparison |
|
@trycoon I am also getting the same error on this one too. :( Basically the crash is happening when too many concurrent http requests are sent to controller. |
|
have you found any solution bro? |
|
@dr3adx no not yet :( |
|
Would this solve any issue? |
gonna test it very soon bro and let u know |
|
I'm on it improving, last published should be ok (and far better), but now
developing TLS with stabilizing the TCP management.
…On Tue, Jun 6, 2023, 21:46 dr3adx ***@***.***> wrote:
Would this solve any issue? https://github.com/HamzaHajeir/H4AsyncTCP
gonna test it very soon bro and let u know
—
Reply to this email directly, view it on GitHub
<#161 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AH3O7J3C7MWH2KO4O46YFN3XJ53GHANCNFSM6AAAAAAVJWOJLI>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
you have Discord bro? |
|
Nope ...
But why?
…On Tue, Jun 6, 2023, 22:11 dr3adx ***@***.***> wrote:
I'm on it improving, last published should be ok (and far better), but now
developing TLS with stabilizing the TCP management.
… <#m_-2325937816652580759_>
On Tue, Jun 6, 2023, 21:46 dr3adx *@*.*> wrote: Would this solve any
issue? https://github.com/HamzaHajeir/H4AsyncTCP
<https://github.com/HamzaHajeir/H4AsyncTCP> gonna test it very soon bro and
let u know — Reply to this email directly, view it on GitHub <#161
(comment)
<#161 (comment)>>,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AH3O7J3C7MWH2KO4O46YFN3XJ53GHANCNFSM6AAAAAAVJWOJLI
<https://github.com/notifications/unsubscribe-auth/AH3O7J3C7MWH2KO4O46YFN3XJ53GHANCNFSM6AAAAAAVJWOJLI>
. You are receiving this because you commented.Message ID: @.*>
you have Discord bro?
—
Reply to this email directly, view it on GitHub
<#161 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AH3O7J57XS7VIOHLBLKAP4LXJ56GHANCNFSM6AAAAAAVJWOJLI>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
|
Hi @HamzaHajeir is your library compatible with ESPAsyncWebServer? |
Hi @johnnytolengo, it's not, really. But the stack comes with a stable H4AsyncWebServer that you can check out. |
You can use this repository with the fix-bugs relevant to memory leaks and crash tcp/ip api calls |
hello bro, I want to try your async TCP library but HTTP library is missing? https://github.com/philbowles/H4AsyncHTTP link is not working, got any idea where can I get async HTTP library? |
|
Hi there, here's the HTTP client library:
https://github.com/HamzaHajeir/ArmadilloHTTP
…On Sun, Nov 26, 2023, 06:25 dr3adx ***@***.***> wrote:
Hi @HamzaHajeir <https://github.com/HamzaHajeir> is your library
compatible with ESPAsyncWebServer?
Hi @johnnytolengo <https://github.com/johnnytolengo>, it's not, really.
But the stack comes with a stable H4AsyncWebServer
<https://github.com/hamzahajeir/h4asyncwebserver> that you can check out.
hello bro, I want to try your async TCP library but HTTP library is
missing? https://github.com/philbowles/H4AsyncHTTP link is not working,
got any idea where can I get async HTTP library?
—
Reply to this email directly, view it on GitHub
<#161 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AH3O7J3WSZ6JNMCM7OK46RLYGKZEHAVCNFSM6AAAAAAVJWOJLKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRWGQ3TQMJZHA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
how do I get this running in platformio? do I need h4plugins ? do I need AardvarkTCP? My current platformio config: I'm trying to compile example program and I get error during linking: https://pastebin.com/ZgStVidc |
Got it. i downloaded your platformio project. However, I can only compile env:esp32s3-devkitc otherwise with other envs i get error: but my board is not esp32s3-devkitc and i cannot upload firmware (even tho it compiles successfully), and it doesnt compile with board i normally use "az-delivery-devkit-v4" |
|
My question is how stable is the H4AsyncWebServer and if worth to replace the ESPAsyncWebServer? As we know the huge problem of the ESPAsyncWebServer is that it crashes very often, and all that crashes are related to the AsyncTCP.h library, isn't it? If I'm right maybe it's time to make an effort all togeather and fix the AsyncTCP.h in order to make it 100% stable without memory leaks and crashes. What's the best fork of the AsyncTCP.h out there? Plan B: switch to H4AsyncWebServer. |
Per the ip_addr_t there's apparently a missing header only for esp32 Per the issue, I've double checked the example, it successfully read the body response: Seems a different API you're requesting? More info is needed to help. |
I personally rely over the H4 stack in my product(s), I can tell you that the environment works as a charm*, including HTTP(s) requests, Webserver (even TLS**), and MQTT over TLS communication, with large payloads. completely async. The testbed was working up to weeks (until I uploaded another sketch or something else), with no memory leak nor crashes. * As long as there's available memory (ESP8266 is so limited). |
I will play around and let you know regarding JSON. But how can I disable logs like H=229600 M=110580 m=219940 S=5596 I tried defining H4AT_DEBUG 0 in config.h and it's not working, they're killing my console :/ I want to disable all logs in your example main.cpp |
These logs were setup for diagnostic, to disable you can comment out these lines: h4.every(300, []()
{
#if defined(ARDUINO_ARCH_ESP32)
Serial.printf("H=%u M=%u m=%u S=%u\n", _HAL_freeHeap(MALLOC_CAP_INTERNAL), _HAL_maxHeapBlock(MALLOC_CAP_INTERNAL), _HAL_minHeapBlock(MALLOC_CAP_INTERNAL), uxTaskGetStackHighWaterMark(NULL));
#else
Serial.printf("H=%u M=%u m=%u\n", _HAL_freeHeap(), _HAL_maxHeapBlock(), _HAL_minHeapBlock());
#endif
h4p["heap"] = _HAL_freeHeap();
h4p["pool"] = mbx::pool.size();
});Also you can comment out any undesired |
Thanks a lot! However, same request works in Postman and returns proper json body. With your H4 weird thing is that status code returned is correct: 200 and content length is 20 (correct), but both asJsonstring & asStdstring methods return empty string. My code & output is here: https://pastebin.com/hVtWtzmj |
|
@HamzaHajeir : I have an app using AsyncTCP, ESPAsyncWebServer and libraries using WebSocket. So I cannot use the H4 stack, but I was wondering if your AsyncTCP fork (https://github.com/HamzaHajeir/H4AsyncTCP) can be directly used as a replacement of the original AsyncTCP instead ? Thanks ! |
Can you insert those two lines and post the output: Serial.printf("data %p len %lu\n", reply.data, reply.length);
dumphex(reply.data, reply.length); |
Hi Matheiu, I'm really cannot tell exactly, there should be some similarity in APIs (onData/onRX) - (onConnect) - (onError). Philbowles had a different internal design. Perhaps this comparison sketch shows some differences in APIs, I think one can try to port the aforementioned libraries to H4AsyncTCP and try out, for TCP stuff it's solid and works great up to my experience and knowledge (spent months to fully support it with fixes, but had to take extreme care dealing with TCP object in upper layers (MQTT/HTTP/Webserver) as it's expirable and nullable pointer that should be nullified on every error / disconnect / connectFail). I'm really unaware of how ESPAsyncWebServer is designed, and whether there's a critical break in dealing to H4AsyncTCP. Take a look at H4AsyncTCP documentation regarding dealing with the server. But perhaps changing to H4AsyncWebServer might be easier, take a look at this comment, beside taking a look on how H4Plugins uses it. Note: Basic authentication was added, and filesystem serving root is fixed (used within H4Plugins) |
This is output: data 0x3ffd47c9 len 0 |
In the file #define ARMA_DEBUG 4Also in ArmadilloHTTP.cpp:198, insert void ArmadilloHTTP::_rx(const uint8_t* d,size_t s){
ARMA_PRINT1("RX 0x%08x len=%d FH=%u\n",d,s,_HAL_freeHeap());
dumphex(d,s);
...What the output would be? |
the json {"error_code":-1 displays now in the log at least! just have to find out why its not being returned in asJson method. |
|
This is because the webserver sends two separate TCP packets, both holding PUSH flag for immediate processing. Other stream-based clients does "suck" data out of pool, unlike H4AsyncTCP which is completely event-driven. Can you checkout Wireshark and validate it's only one TCP packet (not as I've described)? |
|
Alternatively, you can just turn H4AT_DEBUG 2 under H4AT_PRINT2("_raw_recv %p data=%p L=%d f=0x%02x \n",rq,cpydata,cpylen,cpyflags); |
here's the new log: https://pastebin.com/0QnWfkTR |
Yeah, two packets carrying PUSH flag: f=0x01 is PBUF_FLAG_PUSH. The webserver behaviour should be corrected, either to send one TCP packet including body, or to detach PUSH flag from the first TCP packet. |
Impossible to modify webserver, its a commercial device (Tapo P105) by TP-LINK :/ |
@mathieucar Thanks. |
Real problem :, as if they wanted to chunk it out, there's a standardized way to do in HTTP protocol and it's managed within the library*. The issue is similar to this one except TLS isn't used here, we're out of the standards already. I've gave it a thought, if to add a workaround to the library, it wont be a library anymore, as it will be custom to solve that case But what you can do is something will be tough to some extent, is to work with H4AsyncTCP directly, manually write the request, and Alternatively you might use a stream-based HTTP client, as what esp32-arduino offers, which seems to me it solves your current problem if you don't really bother whether it's async or not. * There's a current bug for handling very large responses. |
well its not really a "custom" problem, it works just fine in Postman and other HTTP clients I tried queueing 255 reqs using h4.queue function but not a single req executes :O |
Other HTTP clients are either stream-based, or event-driven. I don't think this is a problem for Stream-based clients to a good extent, but if event driven they might insert the workarounds outside the standards. Some of other clients can do more advanced workarounds to deal with uncompliant servers, as correcting the order of out-of-order HTTP chunks (even worse than that), which happened in my webserver and found the browser just fixes that, before I fixed the bug here ofcourse. If I didn't fix it and you were to request by a moderate client, you will find it very hard to parse, as it will be a bunch of scrambled data to you. Per the current issue, for any TCP packet which holds PUSH flag, it tells you "I'm finished, please process the message immediately", thus the client receives the pushed message and process it as per the HTTP standard, in your case the sender does just lie :).
I'm not sure what you're aiming for, seems want to discover it? The common way to do is via mDNS, can you check the Tapo P105 documentation for such a feature? Per requests, at least some does and the other just don't (you'll need to add callbacks onConnectFail/onError/onDisconnect to know why); but primarily because there's a limit of 16 simultaneous active pcbs (protocol control blocks) in ESP32. |
can 1 pcb only handle 1 simultaneous request at a time? or what does this mean in terms of performance? also device im trying to ping doesnt use mDNS service so i have to http request every IP on subnet (for discovery), how would I go about doing this efficiently with H4? |
|
Each one controls only one TCP connection.
…On Thu, Nov 30, 2023, 17:27 dr3adx ***@***.***> wrote:
well its not really a "custom" problem, it works just fine in Postman and
other HTTP clients
Other HTTP clients are either stream-based, or event-driven. I don't think
this is a problem for Stream-based clients to a good extent, but if event
driven they might insert the workarounds outside the standards. Some of
other clients can do more advanced workarounds to deal with uncompliant
servers, as correcting the order of out-of-order HTTP chunks (even worse
than that), which happened in my webserver and found the browser just fixes
that, before I fixed the bug here ofcourse. If I didn't fix it and you were
to request by a moderate client, you will find it very hard to parse, as it
will be a bunch of scrambled data to you.
Per the current issue, for any TCP packet which holds PUSH flag, it tells
you "I'm finished, please process the message immediately", thus the client
receives the pushed message and process it as per the HTTP standard, in
your case the sender does just lie :).
how can i chain multiple async requests with h4? i need to check every IP
on the subnet, so 255 requests, ill just use status code since json body
isnt working
I tried queueing 255 reqs using h4.queue function but not a single req
executes :O
I'm not sure what you're aiming for, seems want to discover it? The common
way to do is via mDNS, can you check the Tapo P105 documentation for such a
feature?
Per requests, at least some does and the other just don't (you'll need to
add callbacks onConnectFail/onError/onDisconnect to know why); but
primarily because there's a limit of 16 simultaneous active pcbs (protocol
control blocks) in ESP32.
can 1 pcb only handle 1 simultaneous request at a time? or what does this
mean in terms of performance?
—
Reply to this email directly, view it on GitHub
<#161 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AH3O7J2F6FNYAAXLYWNOR63YHCJVDAVCNFSM6AAAAAAVJWOJLKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZTHA4DEMRRGI>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Hi. I'm just reading the thread again. Hope you have managed to solve your issue :-) |
|
FYI I am maintaining since a while now (with some help):
These forks are used in some big projects out there (OpenDTU, Beelance, YaSolR, ESP-DASH, ElegantOTA, ESPConnect, WebSerial, etc). I have an app doing a LOT of websocket calls without any issue (at a rate of about 3-4 push per sec). I've merged inside a couple of PRs standing there, plus added Arduino 3 support, ArduinoJson 7, IPv6 (improved from ESPHome draft), etc You might want to check them out. https://github.com/HamzaHajeir/H4AsyncTCP is not an option because of its too restrictive license. |
Glad to hear you've fixed your issue. Per the license, the author has passed in June 2022, leaving the work in the middle, then I took the lead to complete his work and make it useful to the world, I tried to get to his family (only one sister) regarding the license, and other people tried to reach her, but with no luck, she seems not so engaged on the internet and we have no other contact method. Opening the license to the world is an option if we could not reach her, but it's the last resort for me. BTW I'm running a reliable sketch that publishes 13k of MQTTv5.0 over TLS every 3 seconds, with running HTTPS and WSS, alongwith HTTPs Client that fetches every 15 seconds a TLS HTTP API. |
It would help immensely if you could show us the code that can make high performance multiple simple requests (for an example fetching IP from https://api.ipify.org/?format=json). With high performance i mean sending as many requests as possible on as many tcp connections as possible. Thank you bro. |
Hi @dr3adx, I'm glad to propose this solution for that, I've just developed and uploaded an example for that matter run over the HTTP Client library (ArmadilloHTTP), please take a look: To run it you'll need to use the custom environment, replacing the main.cpp's code with it (Or by using simple |
Happens when I try to send around 6 requests per second. Using ASYNCHttpReqs library. I'm so frustrated I thought async requests will work. Like everything works fine until it crashes couple of minutes later.
AsyncTCP.cpp line 420 specifically
The text was updated successfully, but these errors were encountered: