Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Survey data is not uploaded or stored in CAWS #342

Open
bardram opened this issue Oct 24, 2024 · 7 comments
Open

Survey data is not uploaded or stored in CAWS #342

bardram opened this issue Oct 24, 2024 · 7 comments
Assignees
@bardram
Labels
bug Something isn't working

Comments

@bardram
Copy link
Contributor

bardram commented Oct 24, 2024

Survey data is not available on when downloading data from CAWS.

The only thing available is the completedTask of a survey task, like this:

        {
          "data": {
            "__type": "dk.cachet.carp.completedtask",
            "taskData": {
              "__type": "dk.cachet.carp.survey",
              "surveyResult": {
                "results": {
                  "survey.demographics.1": {
                    "end_date": "2024-10-20T20:39:57.918229",
                    "identifier": "survey.demographics.1",
                    "start_date": "2024-10-20T20:39:57.917076"
                  },
                  "survey.demographics.2": {
                    "end_date": "2024-10-20T20:39:57.918356",
                    "identifier": "survey.demographics.2",
                    "start_date": "2024-10-20T20:39:57.918324"
                  },
                  "survey.demographics.3": {
                    "end_date": "2024-10-20T20:39:57.918410",
                    "identifier": "survey.demographics.3",
                    "start_date": "2024-10-20T20:39:57.918392"
                  }
                },
                "identifier": "demo_survey",
                "start_date": "2024-10-20T20:39:57.913648"
              }
            },
            "taskName": "Task #19"
          },
          "sensorStartTime": 1729449597948697
        },

which does not contain the result of the survey either.

This can be caused by several things:

  1. Data is not uploaded @bardram
  2. Data is not stored @yuanchen233
  3. Data is not extracted in the export file @yuanchen233
@bardram bardram added the bug Something isn't working label Oct 24, 2024
@bardram
Copy link
Contributor Author

bardram commented Oct 24, 2024

Seems like the survey data type is not even part of the data being collected for upload to the data stream service. Here is the list of data type being collected for upload, and survey is not part of this list:

[log] [CAMS DEBUG] CarpDataManager - Starting upload of data batches...
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Primary Phone' and data type 'dk.cachet.carp.heartbeat'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Primary Phone' and data type 'dk.cachet.carp.error'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Primary Phone' and data type 'dk.cachet.carp.triggeredtask'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Primary Phone' and data type 'dk.cachet.carp.completedtask'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Location Service' and data type 'dk.cachet.carp.heartbeat'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Location Service' and data type 'dk.cachet.carp.triggeredtask'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Location Service' and data type 'dk.cachet.carp.completedtask'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Weather Service' and data type 'dk.cachet.carp.heartbeat'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Weather Service' and data type 'dk.cachet.carp.triggeredtask'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Weather Service' and data type 'dk.cachet.carp.completedtask'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Air Quality Service' and data type 'dk.cachet.carp.heartbeat'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Air Quality Service' and data type 'dk.cachet.carp.triggeredtask'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Air Quality Service' and data type 'dk.cachet.carp.completedtask'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Primary Phone' and data type 'dk.cachet.carp.ambientlight'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Primary Phone' and data type 'dk.cachet.carp.stepcount'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Primary Phone' and data type 'dk.cachet.carp.freememory'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Primary Phone' and data type 'dk.cachet.carp.deviceinformation'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Primary Phone' and data type 'dk.cachet.carp.batterystate'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Primary Phone' and data type 'dk.cachet.carp.screenevent'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Primary Phone' and data type 'dk.cachet.carp.activity'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Location Service' and data type 'dk.cachet.carp.location'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Location Service' and data type 'dk.cachet.carp.mobility'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Weather Service' and data type 'dk.cachet.carp.weather'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Air Quality Service' and data type 'dk.cachet.carp.airquality'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Polar HR Sensor' and data type 'dk.cachet.carp.heartbeat'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Polar HR Sensor' and data type 'dk.cachet.carp.triggeredtask'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Polar HR Sensor' and data type 'dk.cachet.carp.completedtask'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Polar HR Sensor' and data type 'dk.cachet.carp.polar.hr'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Polar HR Sensor' and data type 'dk.cachet.carp.polar.ecg'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Primary Phone' and data type 'dk.cachet.carp.currentlocation'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Primary Phone' and data type 'dk.cachet.carp.weather'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Primary Phone' and data type 'dk.cachet.carp.location'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Primary Phone' and data type 'dk.cachet.carp.audio'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Primary Phone' and data type 'dk.cachet.carp.image'.

@bardram
Copy link
Contributor Author

bardram commented Oct 24, 2024

Seems like the survey data type is not even part of the data being collected for upload to the data stream service. Here is the list of data type being collected for upload, and survey is not part of this list:

This is because the survey data type is not part of the list of measures to collect. And this is because it is part of an AppTask and not in the measure list.

@bardram bardram self-assigned this Oct 24, 2024
@bardram
Copy link
Contributor Author

bardram commented Oct 24, 2024

There is also an issue that the JSON uploaded, is the old snake_case JSON from Research Package (and not camelCase). We need to update to the latest version of RP....

@bardram
Copy link
Contributor Author

bardram commented Oct 24, 2024

Survey data also seems to be stored twice in the local DB:

[log] [CAMS DEBUG] SQLiteDataManager - wrote measurement to SQLite - id: 20, type: dk.cachet.carp.survey, device role name: Primary Phone.
[log] [CAMS DEBUG] SQLiteDataManager - wrote measurement to SQLite - id: 21, type: dk.cachet.carp.survey, device role name: Primary Phone.

@bardram
Copy link
Contributor Author

bardram commented Oct 24, 2024 •
edited
Loading

Not all survey data is part of the data downloaded from CAWS.

Note that this is using the "compressed" endpoint.

Here is what I upload as a survey data point:

SURVEY UPLOADED 
{
  "sensorStartTime": 1729776116221468,
  "data": {
    "__type": "dk.cachet.carp.survey",
    "surveyResult": {
      "identifier": "demo_survey",
      "startDate": "2024-10-24T15:21:56.221236",
      "results": {
        "survey.demographics.1": {
          "identifier": "survey.demographics.1",
          "startDate": "2024-10-24T15:21:56.221315",
          "endDate": "2024-10-24T15:21:56.221363",
          "questionTitle": "survey.demographics.question.sex",
          "results": {
            "answer": [
              {
                "__type": "RPChoice",
                "text": "survey.demographics.other",
                "value": 3,
                "isFreeText": false
              }
            ]
          },
          "answerFormat": {
            "__type": "RPChoiceAnswerFormat",
            "questionType": "SingleChoice",
            "choices": [
              {
                "__type": "RPChoice",
                "text": "survey.demographics.femal",
                "value": 1,
                "isFreeText": false
              },
              {
                "__type": "RPChoice",
                "text": "survey.demographics.male",
                "value": 2,
                "isFreeText": false
              },
              {
                "__type": "RPChoice",
                "text": "survey.demographics.other",
                "value": 3,
                "isFreeText": false
              },
              {
                "__type": "RPChoice",
                "text": "survey.demographics.prefer_not",
                "value": 4,
                "isFreeText": false
              }
            ],
            "answerStyle": "SingleChoice"
          }
        },
        "survey.demographics.2": {
          "identifier": "survey.demographics.2",
          "startDate": "2024-10-24T15:21:56.221398",
          "endDate": "2024-10-24T15:21:56.221414",
          "questionTitle": "survey.demographics.question.age",
          "results": {
            "answer": [
              {
                "__type": "RPChoice",
                "text": "70-79",
                "value": 7,
                "isFreeText": false
              }
            ]
          },
          "answerFormat": {
            "__type": "RPChoiceAnswerFormat",
            "questionType": "SingleChoice",
            "choices": [
              {
                "__type": "RPChoice",
                "text": "survey.demographics.under_20",
                "value": 1,
                "isFreeText": false
              },
              {
                "__type": "RPChoice",
                "text": "20-29",
                "value": 2,
                "isFreeText": false
              },
              {
                "__type": "RPChoice",
                "text": "30-39",
                "value": 3,
                "isFreeText": false
              },
              {
                "__type": "RPChoice",
                "text": "40-49",
                "value": 4,
                "isFreeText": false
              },
              {
                "__type": "RPChoice",
                "text": "50-59",
                "value": 5,
                "isFreeText": false
              },
              {
                "__type": "RPChoice",
                "text": "60-69",
                "value": 6,
                "isFreeText": false
              },
              {
                "__type": "RPChoice",
                "text": "70-79",
                "value": 7,
                "isFreeText": false
              },
              {
                "__type": "RPChoice",
                "text": "80-89",
                "value": 8,
                "isFreeText": false
              },
              {
                "__type": "RPChoice",
                "text": "survey.demographics.90_above",
                "value": 9,
                "isFreeText": false
              },
              {
                "__type": "RPChoice",
                "text": "survey.demographics.prefer_not",
                "value": 10,
                "isFreeText": false
              }
            ],
            "answerStyle": "SingleChoice"
          }
        },
        "survey.demographics.3": {
          "identifier": "survey.demographics.3",
          "startDate": "2024-10-24T15:21:56.221436",
          "endDate": "2024-10-24T15:21:56.221453",
          "questionTitle": "survey.demographics.question.smoke",
          "results": {
            "answer": [
              {
                "__type": "RPChoice",
                "text": "survey.demographics.smoke.21+",
                "value": 6,
                "isFreeText": false
              }
            ]
          },
          "answerFormat": {
            "__type": "RPChoiceAnswerFormat",
            "questionType": "SingleChoice",
            "choices": [
              {
                "__type": "RPChoice",
                "text": "survey.demographics.smoke.never",
                "value": 1,
                "isFreeText": false
              },
              {
                "__type": "RPChoice",
                "text": "survey.demographics.smoke.ex",
                "value": 2,
                "isFreeText": false
              },
              {
                "__type": "RPChoice",
                "text": "survey.demographics.smoke.1",
                "value": 3,
                "isFreeText": false
              },
              {
                "__type": "RPChoice",
                "text": "survey.demographics.smoke.1-10",
                "value": 4,
                "isFreeText": false
              },
              {
                "__type": "RPChoice",
                "text": "survey.demographics.smoke.11-20",
                "value": 5,
                "isFreeText": false
              },
              {
                "__type": "RPChoice",
                "text": "survey.demographics.smoke.21+",
                "value": 6,
                "isFreeText": false
              },
              {
                "__type": "RPChoice",
                "text": "survey.demographics.prefer_not",
                "value": 7,
                "isFreeText": false
              }
            ],
            "answerStyle": "SingleChoice"
          }
        }
      }
    }
  }
}

And here is the same data point (check the time stamp in sensorStartTime) when downloaded from CAWS:

SURVEY DOWNLOADED 
        {
          "data": {
            "__type": "dk.cachet.carp.survey",
            "surveyResult": {
              "results": {
                "survey.demographics.1": {
                  "endDate": "2024-10-24T15:21:56.221363",
                  "startDate": "2024-10-24T15:21:56.221315",
                  "identifier": "survey.demographics.1"
                },
                "survey.demographics.2": {
                  "endDate": "2024-10-24T15:21:56.221414",
                  "startDate": "2024-10-24T15:21:56.221398",
                  "identifier": "survey.demographics.2"
                },
                "survey.demographics.3": {
                  "endDate": "2024-10-24T15:21:56.221453",
                  "startDate": "2024-10-24T15:21:56.221436",
                  "identifier": "survey.demographics.3"
                }
              },
              "startDate": "2024-10-24T15:21:56.221236",
              "identifier": "demo_survey"
            }
          },
          "sensorStartTime": 1729776116221468
        },

@bardram
Copy link
Contributor Author

bardram commented Oct 24, 2024

It seems like we are not uploading the full survey data - this is what is uploaded when not using the compressed end point. As can be seen, the survey data is missing.

        {
         "dataStream": {
          "studyDeploymentId": "b8b4bf38-5026-417f-bd2f-218fa704b4f7",
          "deviceRoleName": "Primary Phone",
          "dataType": "dk.cachet.carp.survey"
         },
         "firstSequenceId": 63,
         "measurements": [
          {
           "sensorStartTime": 1729778311577814,
           "data": {
            "__type": "dk.cachet.carp.survey",
            "surveyResult": {
             "identifier": "symptoms_survey",
             "startDate": "2024-10-24T15:58:31.577394",
             "results": {
              "sym_1": {
               "identifier": "sym_1",
               "startDate": "2024-10-24T15:58:31.577492",
               "endDate": "2024-10-24T15:58:31.577541"
              }
             }
            }
           }
          },
          {
           "sensorStartTime": 1729778311577814,
           "data": {
            "__type": "dk.cachet.carp.survey",
            "surveyResult": {
             "identifier": "symptoms_survey",
             "startDate": "2024-10-24T15:58:31.577394",
             "results": {
              "sym_1": {
               "identifier": "sym_1",
               "startDate": "2024-10-24T15:58:31.577492",
               "endDate": "2024-10-24T15:58:31.577541"
              }
             }
            }
           }
          }
         ],
         "triggerIds": [
          10,
          13
         ]
        }

@bardram
Copy link
Contributor Author

bardram commented Oct 24, 2024

It seems like we are not uploading the full survey data - this is what is uploaded when not using the compressed end point.

This was because the result objects in RP - RPResult - was not done using polymorphic serialization. Fixed in v. 1.7.2

This was referenced Oct 24, 2024
Closed
Merged
bardram added a commit that referenced this issue Oct 24, 2024
@bardram
fix of #342
205e834
@bardram bardram mentioned this issue Oct 24, 2024
Merged
bardram added a commit that referenced this issue Oct 24, 2024
@bardram
Bardram/survey upload issue #342 (#343)
bfae9ec 
* removal of uuid from pubspec

* fix of #342
@bardram bardram mentioned this issue Oct 31, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bardram bardram

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@bardram