Bygfiles

A Big file collection manager.

Install

In a virtual env:

python3 -m venv .venv
source .venv/bin/activate
pip install -U pip wheel
pip install git+https://github.com/Terralego/pyfiles@master # Or any last commit
# Then create your setup.py file before using CLI

CLI

Create a settings.py files where you want to execute the cli with this configuration for file storage:

BACKEND = "bygfiles.storages.diskstorage.DiskStorage"

BACKEND_OPTIONS = {
    "basepath": "/tmp/tmpdir",
    "base_url": "http://localhost:8000"
}

And for S3:

BACKEND = "bygfiles.storages.s3storage.S3Storage"

BACKEND_OPTIONS = {
    "access_key":"<you-S3-access-key>",
    "secret_key":"<you-S3-secret-key>",
    "endpoint_url":"<S3-api-endpoint>",
    "region_name":"<region>",
    "bucket_name":"<bucket name>",

}

Then to store a file:

$ bygfiles store <file path> <file.namespace> <file.name> <version>

version should respect the format: YYYY.MM.DD-Rev or any semver like X.Y.Z

To list all version of a file:

$ bygfiles versions <file.namespace> <file.name>

To search for a file:

$ bygfiles search <file.namespace> <file.name> [<version-prefix>]

version-prefix can be YYYY or X or YYYY.MM or X.Y or YYYY.MM.DD or X.Y.Z or Latest. Latest by default if missing.

Finnaly to delete a file:

$ bygfiles delete <file.namespace> <file.name> <version>

To start the web api server:

$ bygfiles serve

Web API

GET on /search/<namespace>/<filename>[?version=<version>]

To get file version download link. Namespace is a namespace to organise data and filename is the file name. You can optionnaly add a version like latest or <year> or <year.month> or <major> or <major>.<minor>, ... You get the latest for the specified version.

GET on /versions/<namespace>/<filename>

To show all avaible file versions.

Python API

See bygfiles.storage classes for more informations.

You can use bygfiles.storage.get_storage(<backend path>, <options>) to initialize your storage.

Features

• An API to download files with rich version selection
• List all version of a file
• Can be used for CSV or Geojson files
• File can have version like 2018.01.10-01
• Find file by a part of the version. 2018 or 2018.01

Roadmap

• Allow authentification with private data
• Handle file diff between versions
• Get the update date of a file to ease caching
• Add a client library and CLI

License

• Free software: MIT license
• Documentation: https://pyfiles.readthedocs.io.

Credits

This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.