Skip to content

Latest commit

 

History

History
92 lines (69 loc) · 10.6 KB

README.md

File metadata and controls

92 lines (69 loc) · 10.6 KB
Raw

The goal of this project is to provide tools for working with large network traffic datasets and to facilitate research in the traffic classification area. The core functions of the cesnet-datazoo package are:

  • A common API for downloading, configuring, and loading of three public datasets of encrypted network traffic.
  • Extensive configuration options for:
    • Selection of train, validation, and test periods.
    • Selection of application classes and splitting classes between known and unknown.
    • Data transformations, such as feature scaling.
  • Built on suitable data structures for experiments with large datasets. There are several caching mechanisms to make repeated runs faster, for example, when searching for the best model configuration.
  • Datasets are offered in multiple sizes to give users an option to start the experiments at a smaller scale (also faster dataset download, disk space, etc.). The default is the S size containing 25 million samples.

🧠 🧠 See a related project CESNET Models providing pre-trained neural networks for traffic classification. 🧠 🧠

📓 📓 Example Jupyter notebooks are included in a separate CESNET Traffic Classification Examples repo. 📓 📓

Datasets

The cesnet-datazoo package currently provides three datasets with details in the following table (you might need to scroll the table horizontally to see all datasets).

  1. CESNET-TLS22
  2. CESNET-QUIC22
  3. CESNET-TLS-Year22
Name CESNET-TLS22 CESNET-QUIC22 CESNET-TLS-Year22
Protocol TLS QUIC TLS
Published in 2022 2023 2023
Collection duration 2 weeks 4 weeks 1 year
Collection period 4.10.2021 - 17.10.2021 31.10.2022 - 27.11.2022 1.1.2022 - 31.12.2022
Application count 191 102 180
Available samples 141392195 153226273 507739073
Available dataset sizes XS, S, M, L XS, S, M, L XS, S, M, L
Cite https://doi.org/10.1016/j.comnet.2022.109467 https://doi.org/10.1016/j.dib.2023.108888 https://doi.org/10.1038/s41597-024-03927-4
Zenodo URL https://zenodo.org/record/7965515 https://zenodo.org/record/7963302 https://zenodo.org/records/10608607
Related papers https://doi.org/10.23919/TMA58422.2023.10199052

Installation

Install the package from pip with:

pip install cesnet-datazoo

or for editable install with:

pip install -e git+https://github.com/CESNET/cesnet-datazoo

Examples

Initialize dataset to create train, validation, and test dataframes

from cesnet_datazoo.datasets import CESNET_QUIC22
from cesnet_datazoo.config import DatasetConfig, AppSelection

dataset = CESNET_QUIC22("/datasets/CESNET-QUIC22/", size="XS")
dataset_config = DatasetConfig(
    dataset=dataset,
    apps_selection=AppSelection.ALL_KNOWN,
    train_period_name="W-2022-44",
    test_period_name="W-2022-45",
)
dataset.set_dataset_config_and_initialize(dataset_config)
train_dataframe = dataset.get_train_df()
val_dataframe = dataset.get_val_df()
test_dataframe = dataset.get_test_df()

The DatasetConfig class handles the configuration of datasets, and calling set_dataset_config_and_initialize initializes train, validation, and test sets with the desired configuration. Data can be read into Pandas DataFrames as shown here or via PyTorch DataLoaders. See CesnetDataset reference.

See more examples in the documentation.

Papers

Acknowledgments

This project was supported by the Ministry of the Interior of the Czech Republic, grant No. VJ02010024: Flow-Based Encrypted Traffic Analysis.