Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Support appendSeedData argument to be an array or array of arrays #18

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 12 additions & 4 deletions lib/model/SeedData.js
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
var log = require("../util/log");
var apiJson = require("../util/api-json");
var apiStream = require("../util/api-stream");
var csv = require("../util/csv");

/**
* @class SeedData
Expand Down Expand Up @@ -50,7 +51,7 @@ SeedData.prototype.readSeedData = function(token, id, callback) {
* @param {StreamReader} outStream - a {@link StreamReader} that will consume the opened HTTP connection
* sending the SeedData down to the client
* @param {function} callback -
* [Called after the API request was finalized]{@link SeedData~appendSeedDataStream(callback)}
* [Called after the API request was finalized]{@link SeedData~readSeedDataStream(callback)}
*/
SeedData.prototype.readSeedDataStream = function(token, id, outStream, callback) {
log.debug("readSeedData(token=" + token + ", id=" + id + ")");
Expand All @@ -59,7 +60,7 @@ SeedData.prototype.readSeedDataStream = function(token, id, outStream, callback)
apiStream(token, seedDataURL, "GET", null, outStream, callback);
};
/**
* @callback SeedData~appendSeedDataStream(callback) callback
* @callback SeedData~readSeedDataStream(callback) callback
*
* @param {?(APIError|Error)} err - {@link null} or an objecrt describing an exception that occured during execution
*/
Expand All @@ -69,15 +70,22 @@ SeedData.prototype.readSeedDataStream = function(token, id, outStream, callback)
*
* @param {string} token - API-Auth token retrieved with {@link Tokens}
* @param {string} id - ID of the Repository Object
* @param {string} content - Content to be appended to the SeedData object, in CSV form
* @param {(string|string[])} content - Content to be appended to the SeedData object, in CSV form or array of strings
* @param {function} callback -
* [Called after the API request was finalized]{@link SeedData~appendSeedData(callback)}
*/
SeedData.prototype.appendSeedData = function(token, id, content, callback) {
log.debug("appendSeedData(token=" + token + ", id=" + id + ", content=" + content + ")");

var seedDataURL = this.getSeedDataURL(id);
apiJson(token, seedDataURL, "POST", content, callback);
csv.toCSV(content, function(err, data) {
if (err) {
callback(err);
return;
}

apiJson(token, seedDataURL, "POST", data, callback);
});
};
/**
* @callback SeedData~appendSeedData(callback) callback
Expand Down
14 changes: 13 additions & 1 deletion lib/util/api-json.js
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,19 @@ function api(token, endpoint, method, requestBody, callback) {
if (responseBody.trim().length === 0) {
responseBody = null;
} else {
responseBody = JSON.parse(responseBody);
try {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a case where api-json.js is being used for a request that doesn't get back JSON?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. CSV data for example when we appendSeedData or readSeedData

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, I think that's a bug in SeedData.readSeedData(). It should be using apiStream instead of apiJson.

There's no concern for SeedData.appendSeedData(), because the response is always empty (204 No Content).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well okay so we are wrapping appendSeedData into the stream and take end and the wrapper for the callback. That's just fine. Remember that you have 2 versions of append/read one stream one json. And one possibly doesn't work currently as it uses the api-json.js instead of api-stream.js.

Additionally using JSON.parse() and hoping to always retrieve JSON feels presumptious. What if the URL we are requesting this from is not an API endpoint, keep in mind the user may point it to another server not from us. What if the Server returns HTML because an Error happend internally and the Servlet was inaccessible? We should guard against these issues better.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But then readSeedData and ReadSeedDataStream are redundant?
Am 22.09.2015 07:35 schrieb "Matt Solnit" [email protected]:

In lib/util/api-json.js
#18 (comment):

@@ -63,7 +63,19 @@ function api(token, endpoint, method, requestBody, callback) {
if (responseBody.trim().length === 0) {
responseBody = null;
} else {

  •                responseBody = JSON.parse(responseBody);
    
  •                try {
    

Actually, I think that's a bug in SeedData.readSeedData(). It should be
using apiStream instead of apiJson.

There's no concern for SeedData.appendSeedData(), because the response is
always empty (204 No Content).


Reply to this email directly or view it on GitHub
https://github.com/SOASTA/repository.js/pull/18/files#r40054154.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like switching to a single readSeedData that just reads a stream, instead of expecting it in JSON.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, so I have a few thoughts on this.

  1. SeedData.readSeedData definitely has a bug, and just as you found, if the response has multiple lines (which I never tested), you'll get SyntaxError: Unexpected string. This is bad.
  2. Removing this method, and requiring the caller to provide a Stream.Writable, is more "pure" but inconvenient IMO.
  3. Having to deal with special edge cases in api-json.js is also bad. We should let JSON.parse() do its thing without having to worry about "expected non-JSON" responses.

I propose that we keep SeedData.readSeedData, but instead of using apiJson(), it instead creates a temporary stream, calls apiStream(), and then "un-wraps" the content, returning it to the caller as a regular string. In fact, it can even use this.readSeedDataStream() with an in-between callback to do the un-wrapping.

Finally, we can add a note to the JSDoc, warning against doing this unless you know the response is small.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds fair.

responseBody = JSON.parse(responseBody);
}
catch(exception) {
if (exception instanceof SyntaxError) {
// if it isn't a SyntaxError because the response was CSV (SeedData)
// do the callback with the Exception. Otherwise move on and handle response
if (!exception.message.match(/^Unexpected token ,$/).hasOwnProperty("length")) {
callback(exception, null);
return;
}
}
}
}

// Consider anything in the 200-299 range to be success.
Expand Down
73 changes: 73 additions & 0 deletions lib/util/csv.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
"use strict";

var csvStringify = require("csv-stringify");

/**
* @module CSV
* @memberof Util
* @desc
* Utility funcitons handling CSV data and Arrays
*/

/**
* @function isArray
*
* @desc
* Test if the passed in argument is an array or not
*
* @param {} object - Argument tested for being an Array or not
*
* @returns {boolean} - true if the argument to the function is an Array
*/
exports.isArray = function(object) {
return Object.prototype.toString.call(object) === "[object Array]";
};

/**
* @namespace csv
* @function is2D
*
* @desc
* Test if the elements in an already identified array are also an arrays
*
* _NOTE:_ This will also fail if only one element is also an array but the rest are singular values!
*
* @param {Object[]} array - An array to test for being a two dimensional array
*
* @returns {boolean} - true if the array is two dimensional
*/
exports.is2D = function(array) {
return array.filter(function(element) { return !exports.isArray(element) }).length === 0;
};

/**
* @namespace csv
* @function toCSV
*
* @desc
* Turn content into CSV formatted string
* Uses {@link csv~is2D} and {@link csv~isArray} to test if the first argument is in the proper format
* otherwise will simply return the content of the first argument, trusting that it is CSV
*
* @param {} content - Any type of object or array that can be turned into a 2D array for processing
* @param {function} callback -
* [Called once stringify has happend or it was determined not to be an array]{@link csv~toCSV(callback)}
*/
exports.toCSV = function(content, callback) {
if(exports.isArray(content)) {
if (!exports.is2D(content)) {
content = [content];
}
csvStringify(content, function(err, csv) {
callback(err, csv);
});
} else {
callback(null, content);
}
}
/**
* @callback csv~toCSV(callback) callback
*
* @param {?Error} error - {@link null} or an Error object describing the error that occured
* @param {?string} csvString - {@link null} in Error case or CSV formatted string of data
*/
1 change: 1 addition & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
"homepage": "https://github.com/SOASTA/repository.js#readme",
"dependencies": {
"commander": "^2.8.1",
"csv-stringify": "0.0.8",
"grunt-jsdoc": "^0.6.7",
"lodash": "^3.10.1",
"winston": "^1.0.1"
Expand Down