Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: output nulls when no cached copy #26

Merged
merged 7 commits into from
Oct 29, 2024
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
docs: instructions for running zipper part of app
simon-20 committed Oct 25, 2024
commit 453a1d9d44e76c3e641e7f9ccfdb1a34cacc0c17
2 changes: 2 additions & 0 deletions .env-example
Original file line number Diff line number Diff line change
@@ -11,6 +11,8 @@ FORCE_REDOWNLOAD_AFTER_HOURS=24

REMOVE_LAST_GOOD_DOWNLOAD_AFTER_FAILING_HOURS=72

ZIP_WORKING_DIR=/tmp/bulk-data-service-zip

# Sample local setup - values read by docker compose (for simple Postgres DB
# creation), and used by the app
DB_NAME=bulk_data_service_db
11 changes: 10 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -59,12 +59,21 @@ docker compose up

The example `.env` file (`.env-example`) is configured to use the above docker compose setup. If you don't use the docker compose setup, then you will need to change the values in the `.env` file accordingly.

Once the docker compose setup is running, start the bulk download app with:
Once the docker compose setup is running, you can run the dataset updater part of the app with (this will download the datasets and upload them to Azurite):

```
dotenv run python src/iati_bulk_data_service.py -- --operation checker --single-run --run-for-n-datasets=50
```

You can run the zipper operation with:

```
dotenv run python src/iati_bulk_data_service.py -- --operation zipper --single-run
```

It will store the ZIP files in the directory defined in the `ZIP_WORKING_DIR` environment variable.


*Note:* not all versions of `dotenv` require a `run` subcommand.

## Development on the app