Cloud requirements

[rhessing]

Love the project, however, I can't find the CPU, memory and disk (IO and size) requirements.

Does anyone know where I find some documentation about these requirements?

[blakebyrnes]

Good question that currently doesn't have a very clear answer. The platform portion of Ulixee is still under very active development, so we don't have any good metrics on what it will take to run in a cluster/private cloud setup yet.

If you're only just running Hero, we've got a number of people running that setup.

Memory
Your memory will be dependent on how many browsers you need to launch for your setup, whether you're using headed, what platform you're on, what websites you load, how much data you are extracting, what types of resources you load, etc, etc. Unfortunately, it is so variable that you often need to test out your workload to see what you need. You can also optimize memory/cpu by reducing the resources loaded on a given page (blockedResourceTypes, blockedResourceUrls), assuming you can still extract under those settings.

Disk Usage

• Each Chrome version you install is currently running at nearly 500mb unzipped, but that varies by OS.
• Hero keeps session replay databases for each session in your OS tmp dir. You can delete these after a session is closed (Session.events.on('closed', ... ), in which case these will not accumulate. Each session stores an entire copy of the data used to build the webpages, so these can get big uncompressed (anywhere from 200kb to 100mb). You can compress them down if you wish to keep them, or delete them if you don't need to troubleshoot anything or replay what happened. They're in tmp, so they'll also clean up on reboot.

CPU
Hero will be mostly memory and network io bound as it waits for Chrome to complete activities. However, there are a number of websites where the activity on the page will cause a cpu to get overwhelmed (oftentimes it's GPU type tasks). There are also still reported cases where concurrent interaction can cause some CPU spikes as it attempts to communicate in parallel with a Chrome instance (this is an open ticket here.

Your best bet (if this question was about Hero) is to try out your workload. There are several users on the Discord channel who have gotten feedback from other users as well (eg)