This is an open source benchmarking tool for Machine Learning practitioners, the purpose of which is to accurately compare multiple Machine Learning algorithms and techniques against one another, under a standardised condition.
Some implemented algorithms use TensorFlow.js, which require you to install the windows-build-tools
for the TFJS binaries to build from Python.
Once downloaded, run npm i
to install the dependencies, execute npm start
to start the server and then visit http://localhost:5000/
.
You will find the controller for loading the various algorithms in ~/server/models/index.js
, this will then load all of the JavaScript files in the ~/server/models/wrappers
directory.
These wrappers are exported classes that require a name, description with a create function. The create
function will be used to return a new instance of a model, which will be ran per each API call, which will return the results.
Also, along with a run
function in the instance, a predictOne
function is also needed, which takes a file buffer as in input to return a prediction of the class.
An example of which would be:
const fs = require("fs");
class exampleWrapper {
constructor() {
this.name = "Algorithm #1";
this.desc = "Lorem ipsum, dolor sit amet consectetur adipisicing elit. Quam vel labore quos sit minus corrupti voluptatibus.";
}
create(imagePercentage, imageAbsPath, params) {
params.classCount = fs.readdirSync(imageAbsPath).length;
return new e(params);
}
}
class e {
constructor(params) {
this.params = params;
}
async predictOne(fileBuffer) {
return "N/A";
}
async run() {
return {
setUpTime: 0,
trainTime: 0,
evaluateTime: 0,
confusionMatrix: Array.from({ length: this.params.classCount }).map(() => Array.from({ length: this.params.classCount }).fill(0)),
accuracy: 0
};
}
}
module.exports = exampleWrapper;
The directory the image datasets should be placed in is ~/server/imageDatasets/datasets/
, which then should have the following folder structure:
datasets
+-- cifar10
| +-- airplane
| | +-- image_x.jpeg
| | +-- image_z.png
| | +-- image_l.png
| +-- automobile
| | +-- image_q.png
| | +-- image_a.jpg
| | +-- image_d.jpeg
+-- MNIST
| +-- 0
| | +-- image_er.png
| | +-- image_do.jpg
| | +-- image_b.jpg
| +-- 1
| | +-- ima_0.jpeg
| | +-- imatg.jpg
You can find the following images datasets by clicking the following links: cifar10 - https://pjreddie.com/projects/cifar-10-dataset-mirror/ MNIST - https://www.kaggle.com/scolianni/mnistasjpg/download
In the various model wrappers there are lines of code to use the gpu version of tfjs, these might require further debugging to install on a local machine, but will increase performance once npm i @tensorflow/tfjs-node-gpu
is ran successfully on a machine.
If you get the following error:
internal/modules/cjs/loader.js:805
return process.dlopen(module, path.toNamespacedPath(filename));
Try rebuilding the package from source (npm rebuild @tensorflow/tfjs-node-gpu --build-from-source
).
Below is our current plan for this project: