Home

Introduction

Welcome to saffron's docs! Saffron was built to help teams collect and compile huge lists of aggregated news and announcement content intuitively and efficiently.

Roadmap

Saffron is currently under development and is soon to be released as a public alpha release. It will soon support distributed data collection by utilizing separate scaping nodes (a.k.a workers), connected together via multiplex peer-2-peer IPFS connections. This will be done to avoid IP or user-agent blocking and make the scraping process more efficient and stable for larger workloads.

Architecture

Saffron's architecture is based on a main node that issues scraping instructions and several worker nodes that do the scraping & upload the data to the offload database.

The term offload database is used here because saffron has 2 separate databases. One for saving the articles it collects - namely offload database- and one to sync the scraping jobs between its worker nodes, namely the grid (because it sounds cool).