Add kobotoolbox source #259

IsaackMwenda · 2023-03-31T11:47:56Z

Initial implementation of Kobo toolbox -> engagement db.

…at_as_urn()

…ices/Engagement-Data-Pipeline into add_kobotoolbox_source

as2388

Thanks @IsaackMwenda, I've reviewed most of this PR now and found small nits only. I'd like to take another skim once we've had our demo and I've had a chance to try Kobo itself myself.

docker-run-kobotoolbox-to-engagement-db.sh

configurations/test_pipeline_configuration.py

src/kobotoolbox_to_engagement_db/kobotoolbox_to_engagement_db.py

src/kobotoolbox_to_engagement_db/configuration.py

IsaackMwenda added 30 commits March 24, 2023 11:25

add kobotoolbox configurations

3fff292

add kobotool docker scripts

7f7f951

Create kobotoolbox_client.py

36fa7e7

Create kobotoolbox_to_engagement_db.py

9e3dfa7

Create sync_kobotoolbox_to_engagement_db.py

5410d27

run kobotoolbox to engdb

81aeaa3

Update run_pipeline.sh

2e0416a

import Logger

089254e

fix minor syntax error

8751a2a

update data_column_name

edb4619

add ignore_invalid_mobile_numbers

011b5eb

add ParticipantIdConfiguration

02fc23f

debug ParticipantIdConfiguration

2fb3da6

rename and import kobotools configuration

02bfe25

add back participant_id_configuration

f548c59

update kobotoolbox_sources availability check

f18dd8a

use json.load

7ca9466

update api token schema

f961a87

isoparse timestamp string when caching.

c8ff426

add debug statements

69d8ca4

add debug statements

7c363ed

Update kobotoolbox_to_engagement_db.py

93e74b9

remove debug scripts

5c997c0

update data_column_name datatype

3b97114

add use kobotoolbox_test db

b958e83

add debug statements

a353720

Handle phone_number with prefix 07

4bf3abf

update docstrings

4450498

Update kobotoolbox_to_engagement_db.py

422c027

add debug statements

1d284e4

IsaackMwenda added 18 commits March 31, 2023 14:43

improve docstrings, logs and simplify _validate_phone_number_and_form…

8888060

…at_as_urn()

Merge branch 'main' into add_kobotoolbox_source

e49ce0f

fix cache paths

769b389

Merge branch 'add_kobotoolbox_source' of https://github.com/AfricasVo…

12748d1

…ices/Engagement-Data-Pipeline into add_kobotoolbox_source

Handle empty responses + Minor code formating

f778b62

Update kobotoolbox_to_engagement_db.py

9ec298c

Update kobotoolbox_client.py

a89ff17

Fix request string when using cache

1b5a5bf

add debug statements

276f66b

convert submitted_after_exclusive to string

72b43a9

update query formart

c340872

use response.content to check for null responses

072709a

Update run_pipeline.sh

11119de

Update run_pipeline.sh

6c0c274

Update docker-run-log-pipeline-event.sh

26d78c9

Update docker-run-log-pipeline-event.sh

514a70b

Minor refactors

1083a8d

Minor formarting

c974246

IsaackMwenda requested review from as2388 and DanielMwendwa April 6, 2023 10:37

as2388 reviewed Apr 17, 2023

View reviewed changes

DanielMwendwa approved these changes May 23, 2023

View reviewed changes

src/kobotoolbox_to_engagement_db/configuration.py Outdated Show resolved Hide resolved

IsaackMwenda and others added 7 commits May 25, 2023 10:00

update kobo testing configurations

59b5a1d

add a process logger

1304e93

update docstrings

a80d8b7

update run_pipeline.sh

6129c4e

update kobotoolbox client

da0ac52

update kobotoolbox to db

2670f86

Merge branch 'main' into add_kobotoolbox_source

f958992

DanielMwendwa merged commit 7fda9ba into main Jul 18, 2024
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add kobotoolbox source #259

Add kobotoolbox source #259

IsaackMwenda commented Mar 31, 2023 •

edited

Loading

as2388 left a comment

Add kobotoolbox source #259

Add kobotoolbox source #259

Conversation

IsaackMwenda commented Mar 31, 2023 • edited Loading

as2388 left a comment

Choose a reason for hiding this comment

IsaackMwenda commented Mar 31, 2023 •

edited

Loading