How can I process a file only if criteria are met in a nodejs express like middleware style?

[bottom-up-ai]

PROBLEM : Packages (multer, formidable, busboy...) usually store the file directly on the disk or memory with a custom validation in options. My problem is that, when I send fields & files at the same time, I don't want to waste resources on processing the entire file on the disk, or memory or even in a external storage like AWS S3, I want to stop everything and return an error to the client.

I would like the following illustrated in the express route above :

1°) Get the file and fields metadata in a multipart request and attach them to the request object

file : size + mimetype
fields : values
2°) Some validation on the properties created from previous step (validation with express-validator)

3°) technologyController.createTechnology which does the following :

If ok : Process the stream of the file & store it directly into AWS S3 (without it being store in memory or in the local disk)
If error : Doesn't process the file & don't create a record in the database but return a list of error to client
I hope my requirements are clear enough, if not don't hesitate.

I can't figure out how to do it properly, it's been days.

Code samples :

multipart.js

const formidable = require('formidable');

exports.getMetadata = (req, res, next) => {

    const form = formidable({
        uploadDir: `${__dirname}/../controllers/uploads`,
        keepExtensions: true,
    });

    const filesInfos = [];

    form.onPart = (part) => {
		// I can filter here based on the mimetype but I don't have the information about the file size
		// The formidable package doesn't include it
		
        if (!part.mimetype) return form._handlePart(part);

        filesInfos.push({ name: part.name, mimetype: part.mimetype })
    };

    form.parse(req, (err, fields, files) => {
		// when parse() is called, files & fields are being processed already
    });

    return next();
};

route.js

const express = require('express');

const technologyController = require('../controllers/technologyController');

const multipart = require('../utils/multipart');
const validation = require('../utils/validation');

const router = express.Router();

router.route('/')
    .post(
        multipart.getMetadata, // get metadata from files & fields on multipart header
        validation.technology, // validation with express-validator module
        technologyController.createTechnology, // validate fields + files (size, mimetype) && create a record in database if ok
    )
;

module.exports = router;

Code samples are available in the public gist I made here : https://gist.github.com/bottom-up-ai/e411f3f7551261d15a93dbe1f459ed80

Thank you for your time and answers ! :)

[GrosSacASac]

There are conflicts in your requests: It is impossible to get all the metadata and also not store anything temporarily.

Formidable is stream focused, so you can only know the file size as the file stream finishes,
if you don't want to upload as it is streaming in you have to store it or else the stream is lost.

Also in the stream fields and files are not ordered.

Imagine getting something https://github.com/node-formidable/formidable/blob/493ec88520c960d0d848d55a929630fde62f2aae/test/fixture/http/encoding/menu_separator.png.http
but times 5, and it is coming character by character

[bottom-up-ai]

Thank you for your reply, I have a better understanding of what's going on behing the scenes.

From my understanding, it seems that the only way I have to validate my fields & files at the same time (same api endpoint) is to save the file and do the validation afterwards.

I don't want to have separate routes for file uploads & fields, as it complicates the logic in my opinion.

May I ask how would you upload a file & fields to the same endpoint while doing validation ?

The logic would be the following :

1. Get the fields & store the file (local disk, remote storage...)
2. Validate the fields & file (size, mimetype)
3. If error on fields validation : return error and delete the saved file
4. If error on file validation : return error and delete the saved file
5. If ok : store fields in database & let the file inside the storage

The problem I had at the beginning was that it's a very inefficient way of processing the data, as I would need to write the entire file somewhere before validation.
Plus, each time I would have a validation error, I would need to process the file again, and again until validation is ok.
If I stream the file directly in a remote storage like AWS S3, I would need to make an API call to delete the file if there's a validation error.

[GrosSacASac]

Yes I would do it like that. Optimize the valid case.

[bottom-up-ai]

Thank you for your answer, I think I have the information I was looking for.

[wbt]

One option you might want to consider is to separate your front-end processing to send only the field data to the backend for some initial processing, then getting an all-clear message to send the rest. You should continue to do robust back-end verification as someone could work around the normal routine, but a two-step process might accelerate the common case (especially if it's common that the criteria aren't met) for good-faith users.

[bottom-up-ai]

Thank you for your answer.

I thought about the 2 steps process initially and I agree that it could make things easier in some ways (for example, handling multipart request only with files and json request with fields).

My problem with this approach is that it kinda "wastes" resources on network request.

Each time you want to register data (fields + file), you make 2 requests instead of 1 (1 for the fields & 1 for the file). You then double the number of API call for each user, for this operation at least.

The second problem I have is, if the fields are sent first and they pass validation, I would need to store them in the database. But what if the file fails validation ? Then I would have the fields saved inside the database without any file associated (while I require the file + the fields).

[wbt]

The right balance really depends on your validation pass rate.

[bottom-up-ai]

I agree.

What I ended up doing is the following (in case someone find that useful).

1. For normal users (they use the application in the expected way) :

If a validation error happens (file size, mimetype...), it will be handled by the front end code first.
It will probably be enough in terms of error handling for them.

2. For malicious users (they are not using the application in the expected way, they try to find vulnerabilities...) :
   It doesn't really matter if the backend validation code is not the "cleanest" possible, in terms of how clean errors are reported to the front end.

For example, I upload on the same endpoint, a file + some fields. But If the file + a field fail the validation, I can't send back the corresponding errors to the front end.
Instead, I can only send one of the 2 errors at the same time. But it doesn't really matter, because if the front end validation is bypassed, that means this is either a malicious user, or someone who has javascript disabled.

Either way, it's not worth the development time, in my humble opinion.