diff --git a/IATI_Data_Flow.drawio.svg b/IATI_Data_Flow.drawio.svg index 1ce3c64..36c9175 100644 --- a/IATI_Data_Flow.drawio.svg +++ b/IATI_Data_Flow.drawio.svg @@ -1,4 +1,1744 @@ - - - -
Unified Platform Data Flow
Unified Platform Data Flow
Refresh
Refresh
Sync Publishers
Sync Publishers
Download Documents
Download Documents
Sync Documents
Sync Documents
Validate
Validate
File Level Schema Check
File Level Schema Check
Validate - Codelist, Ruleset, Activity Level Schema validate
Validate - Codelist, Rul...
1
1
2
2
Safety Valve / Publisher Flag
Safety Valve / Publisher...
Clean
Clean
Copy all valid documents and activities to "clean" container
Copy all valid documents...
Flatten
Flatten
Flatten
Flatten
3
3
Lakify
Lakify
Lakify
Lakify
Solrize
Solrize
Solrize
Solrize
PSQL
PSQLSource XMLClean XML
Solr
SolrLake
File Level Schema Validator API Function
File Level Schema Validator API Function
Full Validator API Function
Full Validator API Function
Flattener API Function
Flattener API Function
1
1
2
2
3
3
Blob Storage
Blob Storage
Database
Database
API
API
Legend
LegendText is not SVG - cannot display \ No newline at end of file + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Unified Platform Data Flow + + + + Unified Platform Data Flow + + + + + + + + + + + Refresh + + + + Refresh + + + + + + + + + + + Sync Publishers + + + + Sync Publishers + + + + + + + + + Download Documents + + + + Download Documents + + + + + + + + + + + Sync Documents + + + + Sync Documents + + + + + + + + + + + Validate + + + + Validate + + + + + + + + + + + File Level Schema Check + + + + File Level Schema Check + + + + + + + + + Validate - Codelist, Ruleset, Activity Level Schema validate + + + + Validate - Codelist, Rul... + + + + + + + + + 1 + + + + 1 + + + + + + + + + 2 + + + + 2 + + + + + + + + + Safety Valve / Publisher Flag + + + + Safety Valve / Publisher... + + + + + + + + + + + + + Clean + + + + Clean + + + + + + + + + + Copy all valid documents and activities to "clean" container + + + + + Copy all valid documents... + + + + + + + + + + + Flatten + + + + Flatten + + + + + + + + + Flatten + + + + Flatten + + + + + + + + + + + Lakify + + + + Lakify + + + + + + + + + Lakify + + + + Lakify + + + + + + + + + + + Solrize + + + + Solrize + + + + + + + + + Solrize + + + + Solrize + + + + + + + + + + PSQL + + + + PSQL + + + + + + + + + + + + + Source XML + + + + + + + + + + + + Clean XML + + + + + + + + + + + Solr + + + + Solr + + + + + + + + + + + Lake + + + + + + + + + + + + File Level Schema Validator API Function + + + + File Level Schema Validator API Function + + + + + + + + + + + Full Validator API Function + + + + Full Validator API Function + + + + + + + + + 1 + + + + 1 + + + + + + + + + 2 + + + + 2 + + + + + + + + + + + Blob Storage + + + + Blob Storage + + + + + + + + + + Database + + + + + Database + + + + + + + + + API + + + + API + + + + + + + + Legend + + + + Legend + + + + + + Text is not SVG - cannot display + + + + diff --git a/README.md b/README.md index 6f04ee5..0a1d3b6 100644 --- a/README.md +++ b/README.md @@ -222,7 +222,9 @@ Service Loop (when container starts) ## Functions -- `main()` - Sends XML to the [iati-flattener](https://github.com/IATI/iati-flattener) which transforms it into a flat JSON document, then stores it in the database (`document.flattened_activities`) in JSONB format. +- `main()` - Flattens XML into a flat JSON document, then stores it in the database (`document.flattened_activities`) in JSONB format. + +Used to use the [iati-flattener service](https://github.com/IATI/iati-flattener), but now it does it using a Python class it the same process. ## Logic