- Progress visualization
- Suspend crawling when meet errors or network issues and can continue later on
- Depth first crawling to maximize parallelism
- Parallelization controlled by thread pool to save memory
- Decoupled connector to control connection behavior
- Assemble crawl logics in a very flexible style like pipelines
- Decouple crawl logics and collected contents
- Debuggability: Automatically collect exceptions during crawling
- Test mode (TODO)