Added new LORIS hook: csv_data_maker

[prioux]

Allows LORIS users to push arbitrary CSV files to CBRAIN/NeuroHub.

Includes some refactoring of code in common with previous LORIS hook.

You can see the main two action methods are short, and much of the code has been moved into the methods create_file_for_request and render_created_file_report.

There is no particular issue associated with this feature, though I did talk to @bryancaron about it this week.

[prioux]

As in PR #994 , here is a small JSON payload you can use to test:

{
   "table_content": [
  [ "line 1", 123,      "hello", "123" ],
  [ "line 2", 43.22,    "this is \" a 'string'", "George O'malley" ],
  [ "line 3", -343.121, "null", "" ]
],
   "result_dp_id": 134,
   "result_group_id": 386,
   "result_filename": "newhook.csv"
}

You'll have to adjust the IDs. Note that if your change "null" to null, the POST operation seems, in Rails, to miss the element in the array. not sure why.

Here is a test script for that JSON:

#!/bin/bash

curl -D - \
  -X POST                                            \
  --data-binary @$1                                  \
  -H 'Accept: application/json'                      \
  -H 'Content-type: application/json'                \
  http://localhost:3000/nh_loris_hooks/csv_data_maker?cbrain_api_token=$2

Invoke it with the json file in first argument and your token in second argument.

[xlecours]

I can agree to that:

  # Receives:
  #  {
  #    table_content:     [ [a, b, c], [d, e, f]... ],
  #    result_filenname:  "hello.csv",
  #    result_dp_id:      123, # optional, can be name
  #    result_group_id:   123, # optional, can be name
  #  }

I haven't check the rail code

[MontrealSergiy]

not critical, but I would call integration services or api hooks rather the loris hooks to encourage other api consumers use that hooks (Obai perhaps?)

[prioux]

I'm not going to change the name. Yes these integrations can be used by non LORIS clients, but they are designed for LORIS.

[MontrealSergiy]

ok. otherwise create files with suggested json as expected. It also adds a timestamp to the name (probably that's what needed).

[MontrealSergiy]

Minor (non-critical) pickings
Invalid data (e.g. invalid) results in html returned rather than only error code (400). If dp server down however returns 503 which is adequate.

[prioux]

@MontrealSergiy The HTML is because you run your server in development mode. In that mode exceptions raised are shown to the developers in a HTML report for debugging. Try again with your server in production mode and you should get a JSON error message instead. There should never be HTML rendered for API calls.

[MontrealSergiy]

oh.. forgot that... retesting. Then another (minor) issue: if the group id is wrong, error code is generated yet file create on default provider/group. Is it intended? ```$ cat data.json
{
"table_content": [
[ "line 1", 123, "hello", "123" ],
[ "line 2", 43.22, "this is " a string", "George O'malley" ],
[ "line 3", -343.121, "null", "" ]
],
"result_dp_id": 11,
"result_group_id": "blabla",
"result_filename": "newhook.csv"
}

sergi@DESKTOP-7IHBN12 MINGW64 ~
$ bash hooktest.sh data.json $API1
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0HTTP/1.1 201 Created
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Location: http://localhost:3000/userfiles/247
Content-Type: application/json; charset=utf-8
ETag: W/"718b826ce0d123a79ff7117311a068d5"
Cache-Control: max-age=0, private, must-revalidate
X-Request-Id: cf9dd495-618e-4d7a-9833-4514a0d64e84
X-Runtime: 0.186696
Transfer-Encoding: chunked

100 502 0 234 100 268 883 1011 --:--:-- --:--:-- --:--:-- 1901{"message":"CSV File created","userfile_id":247,"userfile_name":"newhook.2020-08-25-17:48:46.csv","group_id":20,"group_name":"buser","data_provider_id":4,"data_provider_name":"test1","cbrain_url":"http://localhost:3000/userfiles/247"}

[xlecours]

Suggested change

	# result_filenname: "hello.csv",
	# result_filename: "hello.csv",

[prioux]

Yeah, I've seen these typos, will fix in a future commit

[xlecours]

Suggested change

	# result_filenname: "hello.csv",
	# result_filename: "hello.csv",

[prioux]

@MontrealSergiy Yes, if the data_provider or group selected is wrong, the code substitute something else. Same with filename. So there shoudl never be a crash related to the data provdier ID as long as the first writable online dataprovider that the user sees happens to work.

[MontrealSergiy]

make sense. It might be a bit more confusing with groups. If group id not found the csv file is added to 'user group', which user cannot browse via neurohub (and does not see files in cbrain). Yet it what @xlecours asks, right?