odata_submission_get not working for 1000+ submissions

[adithirgis]

Hello,

odata_submission_get does not download submissions which are more than 1000+ entries.

It shows this error

i Downloading submissions...
Error in curl::curl_fetch_memory(url, handle = handle): URL rejected: Malformed input to a URL function
Request failed [ERROR]. Retrying in 1.5 seconds...
Error in curl::curl_fetch_memory(url, handle = handle): URL rejected: Malformed input to a URL function
Request failed [ERROR]. Retrying in 3.6 seconds...
Error in curl::curl_fetch_memory(url, handle = handle) : 
  URL rejected: Malformed input to a URL function

ODK Central version:

Thanks,
Adithi

[florianm]

Hi and thanks for the bug report!

I'm currently offline and will be able to investigate on my end only from mid October. Of course this is a long wait, so in the meantime could you help me narrow down the issue?

To troubleshoot credentials and connectivity, does any other ruODK function, e.g. project_list(), work?

What happens when you paginate results with the top and skip parameters?

Can you retrieve e.g. just top=10, 100, 1000 submissions?

At what number does it break?

Can you skip=10, 100, 1000 submissions? Sometimes a particular value could break ruODK but this is rare. In some edge cases, Collect, Central, and ruODK disagree on what is a valid value.

[adithirgis]

Hello,

Thanks for the response. I tried using project_list() it works fine and I am able to retrieve all other forms in the project with < 500 submissions. I tried using top = 10, 100 and skip = 10, 100. It shows the same curl error? I want to download submissions in tibble format and OData API is helpful using the odata_submission_get. Should I use the RESTful API? Thank you.

i Downloading submissions...
i Limiting to max 10 records
Error in curl::curl_fetch_memory(url, handle = handle): URL rejected: Malformed input to a URL function
Request failed [ERROR]. Retrying in 1.2 seconds...
Error in curl::curl_fetch_memory(url, handle = handle): URL rejected: Malformed input to a URL function
Request failed [ERROR]. Retrying in 2.8 seconds...
Error in curl::curl_fetch_memory(url, handle = handle) : 
  URL rejected: Malformed input to a URL function

Adithi

[florianm]

There's definitely something amiss. Malformed input to a URL function indicates that we're not sending correct arguments to httr::RETRY (a wrapper to httr::GET).

To debug this I would take the code of odata_submission_get and step line by line (using a debugger or ctrl-enter each line) until I find the offending statement, then inspect the arguments used in that line.

Can you share more about your form? Does the form title contain unusual (non ASCII) characters? Is the form very long or complex? Does it capture lots of media?

[adithirgis]

Hello,

I will do that.
There are no special characters, in the form title. Every other form < 500 submissions works perfectly fine with the arguments. Let me check using the function of odata_submission_get.

Adithi

[florianm]

Hi @adithirgis!

If your error still persists, could you try the following?

# Install the new ruODK release
remotes::install_github(
  "ropensci/ruODK@main",
  dependencies = TRUE,
  upgrade = "always",
  build_vignettes = FALSE
)

library(ruODK)

url <- # your ODK Central URL, no trailing slash, something like "https://ruodk.getodk.cloud"
un <- "your username, keep confidential"
pw <- "your password, keep confidential"

pid <- # Your project ID, e.g. 1
fid <- # your form ID, e.g. "myForm"
table <- "Submissions" # the main submissions table

x <- httr::GET(
    httr::modify_url(
      url,
      path = glue::glue("v1/projects/{pid}/forms/{URLencode(fid, reserved = TRUE)}.svc/{table}")
    ),
    httr::add_headers(Accept = "application/json"),
    httr::authenticate(un, pw)
  ) |>
    httr::content(.)

That's the core of odata_submission_get. What ouput does that generate?