Prerequisites

Make sure you are running on node v13.x and above

Set up

Run npm i from the root of the project

Running the program

There are two ways to run the program

• run crawler in your command line and follow the prompts
• run crawler -u <url> -n <number of workers>
  • the -u flag represents the url to be crawled
  • the -n flag represents the number of workers to be used in crawling the site