PsychicUploadHandler file index, len, and final params not correct

[enriquedbelec]

Hi,
First of all congrats for your hard work on this library. It's the best one I've tested in terms of performance, stability and heap consumption by far. I come from ESPAsyncWebserver.
I'm using the release 1.2.1 of your library.

I want to report some kind of bug I've detected. Maybe I don't know how to use it well.

I want to upload a file to ESP by using jquery ajax request to upload it. So I've used this file handler:

PsychicUploadHandler uploadHandler;
uploadHandler.onUpload([this](PsychicRequest *request, const String& filename, uint64_t index, uint8_t *data, size_t len, bool final) {
	DBLOG_DEBUG("onUpload. filename: %s  index: %d  len: %d  final?: %d", filename.c_str(), index, len, final);
    if (request->hasParam("fileCategory")){
      DBLOG_DEBUG("onUpload. Param fileCategory: %s", request->getParam("fileCategory")->value().c_str());
    }
    if (request->hasParam("fileSize")){
      DBLOG_DEBUG("onUpload. Param fileSize: %s", request->getParam("fileSize")->value().c_str());
    }
    return ESP_OK;
 });
 // Gets called after upload has been handled
 uploadHandler.onRequest([](PsychicRequest *request){
    DBLOG_DEBUG("onRequest");
    return ESP_OK;
 });
 webServer->on("/api/uploadFile", HTTP_POST, &uploadHandler);

And using this javascript client code:

var formData = new FormData();
formData.append('fileCategory', fileCategory);
formData.append('fileSize', fileData.size);
formData.append('file', fileData); // File
$.ajax({
    url: '/api/uploadFile',
    cache: false,
    contentType: false,
    processData: false,
    data: formData,
    headers: {
          'token': window.localStorage.getItem('auth_token'),
    },
    type: 'POST',
    success: function(res) {
		DEBUG_LOG("Success: " + JSON.stringify(res));
         if(successCallback) successCallback(res);
    },
    error: function(err) {
		DEBUG_LOG("Upload error: " + JSON.stringify(err));
        if(errorCallback) errorCallback(err);
    }
});

This is the serial console output while uploading the file:

onUpload. filename: TestFile.zip  index: 1073635472  len: 0  final?: 0
onUpload. Param fileCategory: ota
onUpload. Param fileSize: 118231
onUpload. filename: TestFile.zip  index: 1073635488  len: 8192  final?: 0
onUpload. Param fileCategory: ota
onUpload. Param fileSize: 118231
onUpload. filename: TestFile.zip  index: 7  len: 16384  final?: 0
onUpload. Param fileCategory: ota
onUpload. Param fileSize: 118231
onUpload. filename: TestFile.zip  index: 1073635488  len: 24576  final?: 0
onUpload. Param fileCategory: ota
onUpload. Param fileSize: 118231
onUpload. filename: TestFile.zip  index: 0  len: 32768  final?: 0
onUpload. Param fileCategory: ota
onUpload. Param fileSize: 118231
onUpload. filename: TestFile.zip  index: 7  len: 40960  final?: 0
onUpload. Param fileCategory: ota
onUpload. Param fileSize: 118231
onUpload. filename: TestFile.zip  index: 7  len: 49152  final?: 0
onUpload. Param fileCategory: ota
onUpload. Param fileSize: 118231
onUpload. filename: TestFile.zip  index: 1073635488  len: 57344  final?: 0
onUpload. Param fileCategory: ota
onUpload. Param fileSize: 118231
onUpload. filename: TestFile.zip  index: 1073635488  len: 65536  final?: 0
onUpload. Param fileCategory: ota
onUpload. Param fileSize: 118231
onUpload. filename: TestFile.zip  index: 7  len: 73728  final?: 0
onUpload. Param fileCategory: ota
onUpload. Param fileSize: 118231
onUpload. filename: TestFile.zip  index: 7  len: 81920  final?: 0
onUpload. Param fileCategory: ota
onUpload. Param fileSize: 118231
onUpload. filename: TestFile.zip  index: 0  len: 90112  final?: 0
onUpload. Param fileCategory: ota
onUpload. Param fileSize: 118231
onUpload. filename: TestFile.zip  index: 7  len: 98304  final?: 0
onUpload. Param fileCategory: ota
onUpload. Param fileSize: 118231
onUpload. filename: TestFile.zip  index: 1073635488  len: 106496  final?: 0
onUpload. Param fileCategory: ota
onUpload. Param fileSize: 118231
onUpload. filename: TestFile.zip  index: 7  len: 114688  final?: 0
onUpload. Param fileCategory: ota
onUpload. Param fileSize: 118231
onRequest

What I've see here is that the "index" param is not reporting well the correct index. I think it only should report negative (1073635488 ) only the first time it's invoked the callback, but it repeats the same during the upload. Also the index 0 and 7 I don't know what does it mean.
Second, the "len" para should not repor the "data" param size?
Last one: the "final" param flag should not be changed to true if reached end of file?

The uploading file was working for me in the ESPASyncWebServer fork from mathieucarbou.

Other question: your release 1.2.1 is from August, 14 2024. Will you release new versions?

Thanks everyone for help.

[hoeken]

Yes this does look like a bug. It looks like it might be a sign issue. I'll try to look into this and see what the cause is. I've gotten a bit busy lately but we have a v2 that is under development and is almost ready for release. I will try to get that pushed out by Christmas.

…

On Mon, Dec 2, 2024, 07:21 enriquedbelec ***@***.***> wrote: Hi, First of all congrats for your hard work on this library. It's the best one I've tested in terms of performance, stability and heap consumption by far. I come from ESPAsyncWebserver. I'm using the release 1.2.1 of your library. I want to report some kind of bug I've detected. Maybe I don't know how to use it well. I want to upload a file to ESP by using jquery ajax request to upload it. So I've used this file handler: PsychicUploadHandler uploadHandler; uploadHandler.onUpload([this](PsychicRequest *request, const String& filename, uint64_t index, uint8_t *data, size_t len, bool final) { DBLOG_DEBUG("onUpload. filename: %s index: %d len: %d final?: %d", filename.c_str(), index, len, final); if (request->hasParam("fileCategory")){ DBLOG_DEBUG("onUpload. Param fileCategory: %s", request->getParam("fileCategory")->value().c_str()); } if (request->hasParam("fileSize")){ DBLOG_DEBUG("onUpload. Param fileSize: %s", request->getParam("fileSize")->value().c_str()); } return ESP_OK; }); // Gets called after upload has been handled uploadHandler.onRequest([](PsychicRequest *request){ DBLOG_DEBUG("onRequest"); return ESP_OK; }); webServer->on("/api/uploadFile", HTTP_POST, &uploadHandler); And using this javascript client code: var formData = new FormData(); formData.append('fileCategory', fileCategory); formData.append('fileSize', fileData.size); formData.append('file', fileData); // File $.ajax({ url: '/api/uploadFile', cache: false, contentType: false, processData: false, data: formData, headers: { 'token': window.localStorage.getItem('auth_token'), }, type: 'POST', success: function(res) { DEBUG_LOG("Success: " + JSON.stringify(res)); if(successCallback) successCallback(res); }, error: function(err) { DEBUG_LOG("Upload error: " + JSON.stringify(err)); if(errorCallback) errorCallback(err); } }); This is the serial console output while uploading the file: onUpload. filename: TestFile.zip index: 1073635472 len: 0 final?: 0 onUpload. Param fileCategory: ota onUpload. Param fileSize: 118231 onUpload. filename: TestFile.zip index: 1073635488 len: 8192 final?: 0 onUpload. Param fileCategory: ota onUpload. Param fileSize: 118231 onUpload. filename: TestFile.zip index: 7 len: 16384 final?: 0 onUpload. Param fileCategory: ota onUpload. Param fileSize: 118231 onUpload. filename: TestFile.zip index: 1073635488 len: 24576 final?: 0 onUpload. Param fileCategory: ota onUpload. Param fileSize: 118231 onUpload. filename: TestFile.zip index: 0 len: 32768 final?: 0 onUpload. Param fileCategory: ota onUpload. Param fileSize: 118231 onUpload. filename: TestFile.zip index: 7 len: 40960 final?: 0 onUpload. Param fileCategory: ota onUpload. Param fileSize: 118231 onUpload. filename: TestFile.zip index: 7 len: 49152 final?: 0 onUpload. Param fileCategory: ota onUpload. Param fileSize: 118231 onUpload. filename: TestFile.zip index: 1073635488 len: 57344 final?: 0 onUpload. Param fileCategory: ota onUpload. Param fileSize: 118231 onUpload. filename: TestFile.zip index: 1073635488 len: 65536 final?: 0 onUpload. Param fileCategory: ota onUpload. Param fileSize: 118231 onUpload. filename: TestFile.zip index: 7 len: 73728 final?: 0 onUpload. Param fileCategory: ota onUpload. Param fileSize: 118231 onUpload. filename: TestFile.zip index: 7 len: 81920 final?: 0 onUpload. Param fileCategory: ota onUpload. Param fileSize: 118231 onUpload. filename: TestFile.zip index: 0 len: 90112 final?: 0 onUpload. Param fileCategory: ota onUpload. Param fileSize: 118231 onUpload. filename: TestFile.zip index: 7 len: 98304 final?: 0 onUpload. Param fileCategory: ota onUpload. Param fileSize: 118231 onUpload. filename: TestFile.zip index: 1073635488 len: 106496 final?: 0 onUpload. Param fileCategory: ota onUpload. Param fileSize: 118231 onUpload. filename: TestFile.zip index: 7 len: 114688 final?: 0 onUpload. Param fileCategory: ota onUpload. Param fileSize: 118231 onRequest What I've see here is that the "index" param is not reporting well the correct index. I think it only should report negative (1073635488 ) only the first time it's invoked the callback, but it repeats the same during the upload. Also the index 0 and 7 I don't know what does it mean. Second, the "len" para should not repor the "data" param size? Last one: the "final" param flag should not be changed to true if reached end of file? The uploading file was working for me in the ESPASyncWebServer fork from mathieucarbou. Other question: your release 1.2.1 is from August, 14 2024. Will you release new versions? Thanks everyone for help. — Reply to this email directly, view it on GitHub <#200>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABEHSW2TF5PUMGC7HGDD332DQ7DHAVCNFSM6AAAAABS3FGPECVHI2DSMVQWIX3LMV43ASLTON2WKOZSG4YTCNJVHE4TGOA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[enriquedbelec]

Yes this does look like a bug. It looks like it might be a sign issue. I'll try to look into this and see what the cause is.

Any idea where to find the issue? I've checked the PsychicUploadHandler trying to find it and also tried your V2 dev version, but this bug is still present.

I come from the mathieucarbou fork of ESPAsyncWebServer and there the upload is working ok. I suppose there is something different from this fork on this part.

Thank you.

[mathieucarbou]

@hoeken : the bug is line 110 in v2 in PsychicUploadHandler.cpp

index += received;

received can be negative in case the esp-idf api returns an error code.

I have opened PR #201.

@enriquedbelec : 2 points:

1. you could switch to v2 branch if you can: there are so many improvements, including middleware, that it is worth switching to the dev branch. Note: there are still some potential api breakages that could happen (we have some tasks list in Discord - the project goes really slowly) - I am also part of it,

2. Regarding my ESPAsycnWS fork, if you see some issues of heap issues, please report: so far my testing show that it is using only a few heap. This is probably a matter of wrong config. You can see the perf: https://github.com/mathieucarbou/ESPAsyncWebServer?tab=readme-ov-file#performance. Even if the concurrent request number is not as high as what Psychic can do, we can still run pretty big apps (i do) with only 4K for the async_tcp stack and not a lof os heap. Often, heap usage comes from the wrong API usage, like calling methods with String (which caches the response, etc).

[enriquedbelec]

@hoeken : the bug is line 110 in v2 in PsychicUploadHandler.cpp

index += received;

received can be negative in case the esp-idf api returns an error code.

I have opened PR #201.

Thank you so much for your response and the fix. I'm trying it on the V1 release version.

1. you could switch to v2 branch if you can: there are so many improvements, including middleware, that it is worth switching to the dev branch. Note: there are still some potential api breakages that could happen (we have some tasks list in Discord - the project goes really slowly) - I am also part of it,

Thank you for the suggestion, @mathieucarbou I already tried to use the v2 branch, but the endpoints were not loading/responding on browser. HTML/CSS was not loading like it was on the v1.2.1 release version. Now I'm back to the 1.2.1 with your fix.

2. Regarding my ESPAsycnWS fork, if you see some issues of heap issues, please report: so far my testing show that it is using only a few heap. This is probably a matter of wrong config. You can see the perf: https://github.com/mathieucarbou/ESPAsyncWebServer?tab=readme-ov-file#performance. Even if the concurrent request number is not as high as what Psychic can do, we can still run pretty big apps (i do) with only 4K for the async_tcp stack and not a lof os heap. Often, heap usage comes from the wrong API usage, like calling methods with String (which caches the response, etc).

Your fork of ESPAsyncWebServer is the best by far with this name, but my heap was fragmented a lot when I perform some heavy load of requests after time or simulateneous. This is the reason why I'm trying the PsychicHttp server, but yours is not discarded yet.

[mathieucarbou]

but the endpoints were not loading/responding on browser. HTML/CSS was not loading like it was on the v1.2.1

Maybe this warrants looking into... It might be a bug in v2...

but my heap was fragmented a lot when I perform some heavy load of requests after time or simulateneous.

Ah... yes... Psychic is using esp-idf behind and does not hold buffer of the content that has to be sent, which helps reducing heap fragmentation. That's why when we have access to a pointer to the data to be sent, we have to chose the right methods in ESPAsyncWS to not use any heap buffer (especially use the ones with uint8_t basically).

That's something we plan to change for v5: we have a list of things to so: we'll soon start a v5 of ESPAsycnWS which will only be ESP32 compatible, will have STL string support and better perf and API.

There are also some improvements that could be done where sometimes a buffer is allocated (i.e. for md5 calc) on heap, but on ESP32, these small buffer could be kept on stack.

[enriquedbelec]

@hoeken : the bug is line 110 in v2 in PsychicUploadHandler.cpp

index += received;

received can be negative in case the esp-idf api returns an error code.

I have opened PR #201.

This fix is for the _basicUploadHandler. Do you think it applies the same to the _multipartUploadHandler ? In my case I'm using the multipart one.

[mathieucarbou]

Ah, no this is different... In the multipart there is a difference in the code: _itemSize - _itemBufferIndex, that is the only place I would see an issue... I did not check though.