+ This blog brings you stories about OpenML: why we want to streamline machine learning research, how you can use it, and what we are doing. We are also open to your stories about anything related to open machine learning research, education, and applications. Join the conversation :).
+ + +This blog brings you stories about OpenML: why we want to streamline machine learning research, how you can use it, and what we are doing. We are also open to your stories about anything related to open machine learning research, education, and applications. Join the conversation :).
+ + + +OpenML is a very cool new online platform that aims at improving — as the name says — Open Machine Learning. It stands for Open Data, Open Algorithms and Open Research. OpenML is still in it’s beta phase, but already pretty awesome.
+With this blog post I would like to introduce the main concepts, show who should be interested in the platform and I will go a little into a challenge it faces.
+The following four concepts form the basis of the platform:
+data
task
flow
run
The figure shows how they are connected.
+
You have data that you do not know how to analyse best? Upload your data to OpenML and you will have the whole world helping you. Write a good data and task description to make sure people understand the problem.
+You like taking part in challenges? Being the best solver of a task? Go to OpenML and check out the many tasks and go solve!
+You developed a statistical method or a machine learning algorithm and want to try it out? You will find plenty of data sets and the possibility to make your algorithm public.
+You study statistics, data science, machine learning? You want to know what is out there? On OpenML you will find a wide variety of algorithms and, if the solvers do a good job, info on software and implementation.
+You teach a machine learning class and want the students to participate in a challenge? Make up your own task and let the students try solving it. The platform shows who uploaded what and when.
+There are possibly many other people who will benefit from the platform, like meta analysts, benchmarkers and people I can not think of right now.
+Other than just browsing the website you can access OpenML through quite some interfaces such as R or WEKA. For an example on how to use the R interface check out the tutorial.
+The whole project is of course open source. Check out the different git repositories for all the code and in case you have any complaints.
+Platforms like kaggle, crowdanalytics and innocentive host challenges and give people only part of the data so they can evaluate the performance of the algorithm on a separate data set to (try to) prevent overfitting. So far OpenML does not do that. It always shows all the data, and algorithms are evaluated via resampling procedures (on OpenML called estimation procedures). There are big discussions about how to solve the problem of overfitting on OpenML. They go from keeping part of the data hidden for a certain time in the beginning to doing repeated cross-validation on the (overly) good performing flows on a given task. If you have ideas here, please don’t hesitate to leave me a comment.
+The platform is still in it’s childhood and may not be perfect yet (If you find issues, post them on the github page). But I think it can grow to be a great thing one day.
+Originally published at heidiseibold.github.io on May 2, 2016.
+ + +OpenML stands for Open Machine Learning and is an online platform, which aims at supporting collaborative machine learning online. It is an Open Science project that allows its users to share data, code and machine learning experiments.
+At the time of writing this blog post I am in Eindhoven at an OpenML workshop, where developers and scientists meet to work on improving the project. Some of these people are R users and they (we) are developing an R package that communicates with the OpenML platform.
+
The OpenML R package can list and download data sets and machine learning tasks (prediction challenges). In R one can run algorithms on the these data sets/tasks and then upload the results to OpenML. After successful uploading, the website shows how well the algorithm performs. To run the algorithm on a given task the OpenML R package builds on the mlr package. mlr understands what a task is and can run learners on that task. So all the OpenML package needs to do is convert the OpenML objects to objects mlr understands and then mlr deals with the learning.
+We want to create a little study on the OpenML website, in which we compare different types of Support Vector Machines. The study gets an ID assigned to it, which in our case is 27. We use the function ksvm (with different settings of the function argument type) from package kernlab, which is integrated in mlr (“classif.ksvm”).
+
For details on installing and setting up the OpenML R package please see the guide on GitHub.
+Let’s start conducting the study:
+
library("OpenML") library("mlr") library("farff") library("BBmisc")
+
+dsize = c(100, 500) taskinfo_all = listOMLTasks(number.of.instances = dsize)taskinfo_10cv = subset(taskinfo_all,
+ task.type == "Supervised Classification" &
+ estimation.procedure == "10-fold Crossvalidation" &
+ evaluation.measures == "predictive_accuracy" &
+ number.of.missing.values == 0 &
+ number.of.classes %in% c(2, 4))
+
+taskinfo = taskinfo_10cv[1:3, ]lrn.list = list(
+ makeLearner("classif.ksvm", type = "C-svc"),
+ makeLearner("classif.ksvm", type = "kbb-svc"),
+ makeLearner("classif.ksvm", type = "spoc-svc")
+)grid = expand.grid(task.id = taskinfo$task.id,
+ lrn.ind = seq_along(lrn.list))
+
+runs = lapply(seq_row(grid), function(i) {
+ message(i)
+ task = getOMLTask(grid$task.id[i])
+ ind = grid$lrn.ind[i]
+ runTaskMlr(task, lrn.list[[ind]])
+})## please do not spam the OpenML server by uploading these
+## tasks. I already did that.
+run.id = lapply(runs, uploadOMLRun, tags = "study_27")evals = listOMLRunEvaluations(tag = "study_27")
+
+evals$task.id = as.factor(evals$task.id)
+evals$setup.id = as.factor(evals$setup.id)
+
+library("ggplot2")
+ggplot(evals, aes(x = setup.id, y = predictive.accuracy,
+ color = data.name, group = task.id)) +
+ geom_point() + geom_line()
Now you can go ahead and create a bigger study using the techniques you have learned.
+If you are interested in more, check out the OpenML blog, the paper and the GitHub repos.
+Originally published at mlr-org.github.io.
+ + +October 2019
+Twice a year the OpenML community meets for a workshop/hackathon/unconference. We improve the platform, discuss and learn for 5 days. If that sounds interesting to you, get in touch or follow updates on meet.openml.org.
+This time the workshop took place at Dagstuhl, a great place for Seminars.
+
We had several breakouts where workshop attendees can join to learn, discuss and progress OpenML. In the following we discuss some of the topics we touched.
+Science projects
+Brainstorm on scientific projects to do with OpenML. Prioritize impactful, well-defined research ideas. We came up with quite a long list of very promising research questions that should actually be quite easy to answer based on OpenML. Many of these are along the lines of empirically providing evidence to verify or bust commonly-held beliefs in the community. Frank would gladly hire a strong postdoc (or quite independent PhD student) and maybe a research engineer, to work on these scientific questions under the umbrella of “evidence-based machine learning with OpenML”.
+Benchmarking using OpenML
+Define guidelines on how to define world-class benchmarks and how to run them.
+Diverse datasets
+New and more diverse datasets.
+Dataset quality
+How to measure data quality and how to improve the quality of datasets on OpenML.
+OpenML use cases for novices
+Shortlist common use cases and start writing accessible blog posts for novice users. If you are a OpenML newbie we need your help with this topic.
+The output of this breakout will be at least one blog post. Keep an eye out for them here :)
+Running a competition using OpenML
+We assessed the current issues with running in-class competitions for teaching purposes using OpenML (biggest one: easy use for non-developers) and brainstormed on a new competition format with related competitions, one for each component of a solution, such as HPO, creating good meta-features, creating a good search space, etc.
+Planning future workshops
+Decide on location and timing for the next couple of workshops. The next OpenML workshop will be in Spring (week of March 30th or week of April 14th) close to Munich. For updates check http://meet.openml.org. The workshop will be cohosted with some other Open Source Machine Learning projects.
+Furthermore a workshop in Austin is being planned for next summer and a datathon is in planning. We are planning to organize dev sprints at various PyCons next year.
+New frontend
+Feedback session on new frontend, Additional visualization for datasets
+Future of client APIs
+Currently, a lot of resources are bound developing different client APIs, such as the Python API, R API and Java API. We discussed how we can better share work and code between the different APIs and the server. For now we are working on automatically generating the Swagger API documentation from the PHP function documentation, which in turn will allow us to generate (documented) parts of the APIs, reduce the need for maintenance and will help to spread updates on the API faster.
+Flow 2.0 design
+Current flow design used in OpenML was inspired by Weka, but through time many limitations have been identified, primarily that existing flow does not allow duplicate use of same component and that it cannot express DAG-based ML programs. We started working on a new specification building on insights from mlr3 and d3m projects, centered around DAG representation. Current plan is to prepare a draft specification and implement prototype converters between other systems and this new specification. Once we do that we will re-evaluate the amount of work it took to build those prototypes and how well the specification satisfied those other systems.
+Random Bot
+The LRZ in Munich provided us with CPU time during the SuperMUC-NG supercomputer test phase, which we used to perform experiments of popular machine learning algorithms with random hyperparameter configurations. This resulted in millions of data points on more than a hundred datasets that we will analyse and publish. The data can be used to learn about typical behaviour of different learners across different datasets, and to construct surrogate models for tuning algorithm benchmarks.
+AutoML Benchmark (Janek)
+A study was created containing 76 binary and multiclass tasks of reasonable difficulty.
+These can be used as a more difficult version of OpenML-100 or in amlb a platform for reproducible benchmarking of AutoML systems.
+R API
+Short session on how the R api will (need to) change. The main issue discussed was that the OpenML R package runs with mlr and breaks when the new package (mlr3) is loaded. We will update the current OpenML R package to work with mlr3. At the same time we are thinking about a vision for a rewrite of the OpenML R package.
+Python API
+We made a lot of improvements to the Python API over the week, with over 20 PRs merged! We’ve added more examples on how to use the package, fixed bugs, improved documentation and refactored code. In the coming days we’re going to make all these improvements available in a new PyPI release. For those looking for a higher level overview of the package, we will publish a paper next week which highlights use-cases, its software design, and project structure.
+Benchmarking paper
+We are working on a comprehensive paper using sklearn, mlr and WEKA, which should demonstrate how OpenML can be used for proper benchmarking and analysis.
+Guidelines / Overfitting / Comparable Metalearning
+There are plans for writing a guidelines and pitfalls paper on benchmarking, meta-overfitting and statistical analysis of results on OpenML.
+Data Formats
+Currently OpenML supports only tabular data in ARFF data format. This is very limiting for many ML tasks. We discussed and explored other data formats we could use as the future next data format. We will post a separate blog post about our process and insights.
+Funding
+OpenML is looking for funding (developers). New ideas on obtaining funding are very welcome. We discussed some ideas: American funding (we need a collaboration partner); ALICE / CLAIRE; Companies. We are a foundation now, which might make it easier.
+Some of the PIs (in particular Bernd Bischl, Frank Hutter and Dawn Song) in the project offer positions with a mix of ML research and development. Contact them if you are interested!
+
We had some talks at the workshop as well:
+Mitar Milutinovic: A short introduction to Data Driven Discovery (D3M)
Yiwen Zhu and Markus Weimer: Large-scale analysis of Jupyter notebooks
Martin Binder, Michel Lang, Florian Pfisterer, Bernd Bischl: Pipelining with mlr3
… and lots of fun…
+
Wanna join the OpenML community? Get in touch!
+ + + +OpenML is an online Machine Learning (ML) experiments database accessible to everyone for free. The core idea is to have a single repository of datasets and results of ML experiments on them. Despite having gained a lot of popularity in recent years, with a plethora of tools now available, the numerous ML experimentations continue to happen in silos and not necessarily as one whole shared community. In this post, we shall try to get a brief glimpse of what OpenML offers and how it can fit our current Machine Learning practices.
+Let us jump straight at getting our hands dirty by building a simple machine learning model. If it is simplicity we are looking for, it has to be the Iris dataset that we shall work with. In the example script below, we are going to load the Iris dataset available with scikit-learn, use 10-fold cross-validation to evaluate a Random Forest of 10 trees. Sounds trivial enough and is indeed less than 10 lines of code.
+from sklearn import datasets
+from sklearn.svm import SVC
+from sklearn.ensemble import RandomForestClassifier
+from sklearn.model_selection import cross_val_score# Loading Iris dataset
+X, y = datasets.load_iris(return_X_y=True)
+print(X.shape, y.shape)(150, 4) (150,)# Initializing a Random Forest with
+# arbitrary hyperparameters
+# max_depth kept as 2 since Iris has
+# only 4 features
+clf = RandomForestClassifier(n_estimators=10, max_depth=2)scores = cross_val_score(clf, X, y, cv=5, scoring='accuracy')
+print("Mean score : {:.5f}".format(scores.mean()))Mean score : 0.94000A simple script and we achieve a mean accuracy of 95.33%. That was easy. It is really amazing how far we have come with ML tools that make it easy to get started. As a result, we have hundreds of thousands of people working with these tools every day. That inevitably leads to the reinvention of the wheel. The tasks that each individual ML practitioner performs often have significant overlaps and can be omitted by reusing what someone from the community has done already. At the end of the day, we didn’t build a Random Forest model all the way from scratch. We gladly reused code written by generous folks from the community. The special attribute of our species is the ability to work as a collective wherein our combined intellect becomes larger than the individual sum of parts. Why not do the same for ML? I mean, can I see what other ML practitioners have done to get better scores on the Iris dataset?
+Answering this is one of the targets of this post. We shall subsequently explore if this can be done, with the help of OpenML. However, first, we shall briefly familiarize ourselves with few terminologies and see how we can split the earlier example we saw into modular components.
+
Image source: https://medium.com/open-machine-learning/openml-1e0d43f0ae13
+Dataset: OpenML houses over 2k+ active datasets for various regression, classification, clustering, survival analysis, stream processing tasks and more. Any user can upload a dataset. Once uploaded, the server computes certain meta-features on the dataset - Number of classes, Number of missing values, Number of features, etc. With respect to our earlier example, the following line is the equivalent of fetching a dataset from OpenML.
+X, y = datasets.load_iris(return_X_y=True)Task: A task is linked to a specific dataset, defining what the target/dependent variable is. Also specifies evaluation measures such as - accuracy, precision, area under curve, etc. or the kind of estimation procedure to be used such as - 10-fold cross-validation, n% holdout set, etc. With respect to our earlier example, the parameters to the following function call capture the idea of a task.
+scores = cross_val_score(clf, X, y, cv=5, scoring='accuracy')Flow: Describes the kind of modelling to be performed. It could be a flow or a series of steps, i.e., a scikit-learn pipeline. For now, we have used a simple Random Forest model which is the flow component here.
+clf = RandomForestClassifier(n_estimators=10, max_depth=2)Run: Pairs a flow and task together which results in a run. The run has the predictions which are turned into evaluations by the server. This is effectively captured by the execution of the line:
+scores = cross_val_score(clf, X, y, cv=5, scoring='accuracy')Now, this may appear a little obfuscating given that we are trying to compartmentalize a simple 10-line code which works just fine. However, if we take a few seconds to go through the 4 components explained above, we can see that it makes our training of a Random Forest on Iris a series of modular tasks. Modules are such a fundamental concept in Computer Science. They are like Lego blocks. Once we have modules, it means we can plug and play at ease. The code snippet below attempts to rewrite the earlier example using the ideas of the OpenML components described, to give a glimpse of what we can potentially gain during experimentations.
+from sklearn import datasets
+from sklearn.svm import SVC
+from sklearn.ensemble import RandomForestClassifier
+from sklearn.model_selection import cross_val_score# To load IRIS dataset as a dataset module/component
+def dataset():
+ X, y = datasets.load_iris(return_X_y=True)
+ return X, y# Tasks here define the number of cross-validation folds
+# and the scoring metric to be used for evaluation
+def task_1(f):
+ X, y = dataset() # loads IRIS
+ return cross_val_score(f, X, y, cv=5,
+ scoring='accuracy')
+
+def task_2(f):
+ X, y = dataset() # loads IRIS
+ return cross_val_score(f, X, y, cv=15,
+ scoring='balanced_accuracy')# Flows determine the modelling technique to be applied
+# Helps define a model irrespective of dataset or tasks
+def flow_1():
+ clf = RandomForestClassifier(n_estimators=10, max_depth=2)
+ return clf
+
+def flow_2():
+ clf = SVC(gamma='auto', kernel='linear')
+ return clf # Runs essentially evaluates a task-flow pairing
+# and therefore in effect executs the modelling
+# of a dataset as per the task task definition
+def run(task, flow):
+ return task(flow)# Results for Random Forest
+rf_task_1 = run(task_1, flow_1())
+rf_task_2 = run(task_2, flow_1())
+print("RF using task 1: {:<.5}; task 2: {:<.5}".format(rf_task_1.mean(), rf_task_2.mean()))
+
+# Results for SVM
+svm_task_1 = run(task_1, flow_2())
+svm_task_2 = run(task_2, flow_2())
+print("SVM using task 1: {:<.5}; task 2: {:<.5}".format(svm_task_1.mean(), svm_task_2.mean()))RF using task 1: 0.95333; task 2: 0.94444
+SVM using task 1: 0.98; task 2: 0.97222We can, therefore, compose various different tasks, flows, which are independent operations. Runs can then pair any such task and flow to construct an ML workflow and return the evaluated scores. This approach can help us define such components one-time, and we can extend this for any combination of a dataset, model, and for any number of evaluations in the future. Imagine if the entire ML community defines such tasks and various simple to complicated flows that they use in their daily practice. We can build custom working ML pipeline and even get to compare performances of our techniques on the same task with others! OpenML aims exactly for that. In the next section of this post, we shall scratch the surface of OpenML to see if we can actually do with OpenML what it promises.
+OpenML-Python can be installed using pip or by cloning the git repo and installing the current development version. So shall we then install OpenML? ;) It will be beneficial if the code snippets are tried out as this post is read. A consolidated Jupyter notebook with all the code can be found here.
+Now that we have OpenML, let us jump straight into figuring out how we can get the Iris dataset from there. We can always browse theOpenML website and search for Iris. That is the easy route. Let us get familiar with the programmatic approach and learn how to fish instead. The OpenML-Python API can be found here.
+In the example below, we will list out all possible datasets available in OpenML. We can choose the output format. I’ll go with dataframe so that we obtain a pandas DataFrame and can get a neat tabular representation to search and sort specific entries.
+import openml
+
+import numpy as np
+import pandas as pd# Fetching the list of all available datasets on OpenML
+d = openml.datasets.list_datasets(output_format='dataframe')
+print(d.shape)
+
+# Listing column names or attributes that OpenML offers
+for name in d.columns:
+ print(name)(3073, 16)
+did
+name
+version
+uploader
+status
+format
+MajorityClassSize
+MaxNominalAttDistinctValues
+MinorityClassSize
+NumberOfClasses
+NumberOfFeatures
+NumberOfInstances
+NumberOfInstancesWithMissingValues
+NumberOfMissingValues
+NumberOfNumericFeatures
+NumberOfSymbolicFeaturesprint(d.head()) did name version uploader status format MajorityClassSize \
+2 2 anneal 1 1 active ARFF 684.0
+3 3 kr-vs-kp 1 1 active ARFF 1669.0
+4 4 labor 1 1 active ARFF 37.0
+5 5 arrhythmia 1 1 active ARFF 245.0
+6 6 letter 1 1 active ARFF 813.0
+
+ MaxNominalAttDistinctValues MinorityClassSize NumberOfClasses \
+2 7.0 8.0 5.0
+3 3.0 1527.0 2.0
+4 3.0 20.0 2.0
+5 13.0 2.0 13.0
+6 26.0 734.0 26.0
+
+ NumberOfFeatures NumberOfInstances NumberOfInstancesWithMissingValues \
+2 39.0 898.0 898.0
+3 37.0 3196.0 0.0
+4 17.0 57.0 56.0
+5 280.0 452.0 384.0
+6 17.0 20000.0 0.0
+
+ NumberOfMissingValues NumberOfNumericFeatures NumberOfSymbolicFeatures
+2 22175.0 6.0 33.0
+3 0.0 0.0 37.0
+4 326.0 8.0 9.0
+5 408.0 206.0 74.0
+6 0.0 16.0 1.0 The column names indicate that they contain the meta-information about each of the datasets, and at this instance, we have access to 2958 datasets as indicated by the shape of the dataframe. We shall try searching for ‘iris’ in the column name and also use the version column to sort the results.
+# Filtering dataset list to have 'iris' in the 'name' column
+# then sorting the list based on the 'version'
+d[d['name'].str.contains('iris')].sort_values(by='version').head()| + | did | +name | +version | +uploader | +status | +format | +MajorityClassSize | +MaxNominalAttDistinctValues | +MinorityClassSize | +NumberOfClasses | +NumberOfFeatures | +NumberOfInstances | +NumberOfInstancesWithMissingValues | +NumberOfMissingValues | +NumberOfNumericFeatures | +NumberOfSymbolicFeatures | +
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 61 | +61 | +iris | +1 | +1 | +active | +ARFF | +50.0 | +3.0 | +50.0 | +3.0 | +5.0 | +150.0 | +0.0 | +0.0 | +4.0 | +1.0 | +
| 41950 | +41950 | +iris_test_upload | +1 | +4030 | +active | +ARFF | +50.0 | +3.0 | +50.0 | +3.0 | +5.0 | +150.0 | +0.0 | +0.0 | +4.0 | +1.0 | +
| 42261 | +42261 | +iris-example | +1 | +348 | +active | +ARFF | +50.0 | +NaN | +50.0 | +3.0 | +5.0 | +150.0 | +0.0 | +0.0 | +4.0 | +1.0 | +
| 451 | +451 | +irish | +1 | +2 | +active | +ARFF | +278.0 | +10.0 | +222.0 | +2.0 | +6.0 | +500.0 | +32.0 | +32.0 | +2.0 | +4.0 | +
| 969 | +969 | +iris | +3 | +2 | +active | +ARFF | +100.0 | +2.0 | +50.0 | +2.0 | +5.0 | +150.0 | +0.0 | +0.0 | +4.0 | +1.0 | +
Okay, so the iris dataset with the version as 1 has an ID of 61. For verification, we can check the website for dataset ID 61. We can see that it is the original Iris dataset which is of interest to us - 3 classes of 50 instances, with 4 numeric features. However, we shall retrieve the same information, as promised, programmatically.
+iris = openml.datasets.get_dataset(61)
+irisOpenML Dataset
+==============
+Name..........: iris
+Version.......: 1
+Format........: ARFF
+Upload Date...: 2014-04-06 23:23:39
+Licence.......: Public
+Download URL..: https://www.openml.org/data/v1/download/61/iris.arff
+OpenML URL....: https://www.openml.org/d/61
+# of features.: 5
+# of instances: 150iris.features{0: [0 - sepallength (numeric)],
+ 1: [1 - sepalwidth (numeric)],
+ 2: [2 - petallength (numeric)],
+ 3: [3 - petalwidth (numeric)],
+ 4: [4 - class (nominal)]}print(iris.description)**Author**: R.A. Fisher
+**Source**: [UCI](https://archive.ics.uci.edu/ml/datasets/Iris) - 1936 - Donated by Michael Marshall
+**Please cite**:
+
+**Iris Plants Database**
+This is perhaps the best known database to be found in the pattern recognition literature. Fisher's paper is a classic in the field and is referenced frequently to this day. (See Duda & Hart, for example.) The data set contains 3 classes of 50 instances each, where each class refers to a type of iris plant. One class is linearly separable from the other 2; the latter are NOT linearly separable from each other.
+
+Predicted attribute: class of iris plant.
+This is an exceedingly simple domain.
+
+### Attribute Information:
+ 1. sepal length in cm
+ 2. sepal width in cm
+ 3. petal length in cm
+ 4. petal width in cm
+ 5. class:
+ -- Iris Setosa
+ -- Iris Versicolour
+ -- Iris VirginicaWith the appropriate dataset available, let us briefly go back to the terminologies we discussed earlier. We have only used the dataset component so far. The dataset component is closely tied with the task component. To reiterate, the task would describe how the dataset will be used.
+We shall firstly list all available tasks that work with the Iris dataset. However, we are only treating Iris as a supervised classification problem and hence will filter accordingly. Following which, we will collect only the task IDs of the tasks relevant to us.
+df = openml.tasks.list_tasks(data_id=61, output_format='dataframe')
+df.head()| + | tid | +ttid | +did | +name | +task_type | +status | +estimation_procedure | +evaluation_measures | +source_data | +target_feature | +... | +NumberOfFeatures | +NumberOfInstances | +NumberOfInstancesWithMissingValues | +NumberOfMissingValues | +NumberOfNumericFeatures | +NumberOfSymbolicFeatures | +number_samples | +cost_matrix | +quality_measure | +target_value | +
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 59 | +59 | +1 | +61 | +iris | +Supervised Classification | +active | +10-fold Crossvalidation | +predictive_accuracy | +61 | +class | +... | +5 | +150 | +0 | +0 | +4 | +1 | +NaN | +NaN | +NaN | +NaN | +
| 118 | +118 | +3 | +61 | +iris | +Learning Curve | +active | +10 times 10-fold Learning Curve | +predictive_accuracy | +61 | +class | +... | +5 | +150 | +0 | +0 | +4 | +1 | +4 | +NaN | +NaN | +NaN | +
| 289 | +289 | +1 | +61 | +iris | +Supervised Classification | +active | +33% Holdout set | +predictive_accuracy | +61 | +class | +... | +5 | +150 | +0 | +0 | +4 | +1 | +NaN | +NaN | +NaN | +NaN | +
| 1758 | +1758 | +3 | +61 | +iris | +Learning Curve | +active | +10-fold Learning Curve | +predictive_accuracy | +61 | +class | +... | +5 | +150 | +0 | +0 | +4 | +1 | +4 | +NaN | +NaN | +NaN | +
| 1823 | +1823 | +1 | +61 | +iris | +Supervised Classification | +active | +5 times 2-fold Crossvalidation | +predictive_accuracy | +61 | +class | +... | +5 | +150 | +0 | +0 | +4 | +1 | +NaN | +NaN | +NaN | +NaN | +
5 rows × 24 columns
+# Filtering only the Supervised Classification tasks on Iris
+df.query("task_type=='Supervised Classification'").head()| + | tid | +ttid | +did | +name | +task_type | +status | +estimation_procedure | +evaluation_measures | +source_data | +target_feature | +... | +NumberOfFeatures | +NumberOfInstances | +NumberOfInstancesWithMissingValues | +NumberOfMissingValues | +NumberOfNumericFeatures | +NumberOfSymbolicFeatures | +number_samples | +cost_matrix | +quality_measure | +target_value | +
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 59 | +59 | +1 | +61 | +iris | +Supervised Classification | +active | +10-fold Crossvalidation | +predictive_accuracy | +61 | +class | +... | +5 | +150 | +0 | +0 | +4 | +1 | +NaN | +NaN | +NaN | +NaN | +
| 289 | +289 | +1 | +61 | +iris | +Supervised Classification | +active | +33% Holdout set | +predictive_accuracy | +61 | +class | +... | +5 | +150 | +0 | +0 | +4 | +1 | +NaN | +NaN | +NaN | +NaN | +
| 1823 | +1823 | +1 | +61 | +iris | +Supervised Classification | +active | +5 times 2-fold Crossvalidation | +predictive_accuracy | +61 | +class | +... | +5 | +150 | +0 | +0 | +4 | +1 | +NaN | +NaN | +NaN | +NaN | +
| 1939 | +1939 | +1 | +61 | +iris | +Supervised Classification | +active | +10 times 10-fold Crossvalidation | +predictive_accuracy | +61 | +class | +... | +5 | +150 | +0 | +0 | +4 | +1 | +NaN | +NaN | +NaN | +NaN | +
| 1992 | +1992 | +1 | +61 | +iris | +Supervised Classification | +active | +Leave one out | +predictive_accuracy | +61 | +class | +... | +5 | +150 | +0 | +0 | +4 | +1 | +NaN | +NaN | +NaN | +NaN | +
5 rows × 24 columns
+# Collecting all relevant task_ids
+tasks = df.query("task_type=='Supervised Classification'")['tid'].to_numpy()
+print(len(tasks))13That settles the task component too. Notice how for one dataset (61), we obtain 11 task IDs which are of interest to us. This should illustrate the one-to-many relationship that dataset-task components can have. We have 2 more components to explore - flows, runs. We could list out all possible flows and filter out the ones we want, i.e., Random Forest. However, let us instead fetch all the evaluations made on the Iris dataset using the 11 tasks we collected above.
+We shall subsequently work with the scikit-learn based task which has been uploaded/used the most. We shall then further filter out the list of evaluations from the selected task (task_id=59 in this case), depending on if Random Forest was used.
+# Listing all evaluations made on the 11 tasks collected above
+# with evaluation metric as 'predictive_accuracy'
+task_df = openml.evaluations.list_evaluations(function='predictive_accuracy', task=tasks, output_format='dataframe')
+task_df.head()| + | run_id | +task_id | +setup_id | +flow_id | +flow_name | +data_id | +data_name | +function | +upload_time | +uploader | +uploader_name | +value | +values | +array_data | +
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | +81 | +59 | +12 | +67 | +weka.BayesNet_K2(1) | +61 | +iris | +predictive_accuracy | +2014-04-07 00:05:11 | +1 | +janvanrijn@gmail.com | +0.940000 | +None | +None | +
| 1 | +161 | +59 | +13 | +70 | +weka.SMO_PolyKernel(1) | +61 | +iris | +predictive_accuracy | +2014-04-07 00:55:32 | +1 | +janvanrijn@gmail.com | +0.960000 | +None | +None | +
| 2 | +234 | +59 | +1 | +56 | +weka.ZeroR(1) | +61 | +iris | +predictive_accuracy | +2014-04-07 01:33:24 | +1 | +janvanrijn@gmail.com | +0.333333 | +None | +None | +
| 3 | +447 | +59 | +6 | +61 | +weka.REPTree(1) | +61 | +iris | +predictive_accuracy | +2014-04-07 06:26:27 | +1 | +janvanrijn@gmail.com | +0.926667 | +None | +None | +
| 4 | +473 | +59 | +18 | +77 | +weka.LogitBoost_DecisionStump(1) | +61 | +iris | +predictive_accuracy | +2014-04-07 06:39:27 | +1 | +janvanrijn@gmail.com | +0.946667 | +None | +None | +
# Filtering based on sklearn (scikit-learn)
+task_df = task_df[task_df['flow_name'].str.contains("sklearn")]
+task_df.head()| + | run_id | +task_id | +setup_id | +flow_id | +flow_name | +data_id | +data_name | +function | +upload_time | +uploader | +uploader_name | +value | +values | +array_data | +
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 144 | +1849043 | +59 | +29015 | +5500 | +sklearn.ensemble.forest.RandomForestClassifier... | +61 | +iris | +predictive_accuracy | +2017-03-03 17:10:12 | +1 | +janvanrijn@gmail.com | +0.946667 | +None | +None | +
| 145 | +1853409 | +59 | +30950 | +5873 | +sklearn.pipeline.Pipeline(Imputer=openml.utils... | +61 | +iris | +predictive_accuracy | +2017-03-21 22:08:01 | +1 | +janvanrijn@gmail.com | +0.960000 | +None | +None | +
| 146 | +6130126 | +59 | +4163633 | +7108 | +sklearn.model_selection._search.RandomizedSear... | +61 | +iris | +predictive_accuracy | +2017-08-21 11:07:40 | +1 | +janvanrijn@gmail.com | +0.960000 | +None | +None | +
| 147 | +6130128 | +59 | +4163634 | +7108 | +sklearn.model_selection._search.RandomizedSear... | +61 | +iris | +predictive_accuracy | +2017-08-21 11:08:06 | +1 | +janvanrijn@gmail.com | +0.946667 | +None | +None | +
| 148 | +6715383 | +59 | +4747289 | +7117 | +sklearn.model_selection._search.RandomizedSear... | +61 | +iris | +predictive_accuracy | +2017-09-01 02:56:44 | +1 | +janvanrijn@gmail.com | +0.960000 | +None | +None | +
# Counting frequency of the different tasks used to
+# solve Iris as a supervised classification using scikit-learn
+task_df['task_id'].value_counts()59 1985
+10107 25
+289 1
+Name: task_id, dtype: int64# Retrieving the most used task
+t = openml.tasks.get_task(59)
+tOpenML Classification Task
+==========================
+Task Type Description: https://www.openml.org/tt/1
+Task ID..............: 59
+Task URL.............: https://www.openml.org/t/59
+Estimation Procedure.: crossvalidation
+Evaluation Measure...: predictive_accuracy
+Target Feature.......: class
+# of Classes.........: 3
+Cost Matrix..........: Available# Filtering for only task_id=59
+task_df = task_df.query("task_id==59")# Filtering based on Random Forest
+task_rf = task_df[task_df['flow_name'].str.contains("RandomForest")]
+task_rf.head()| + | run_id | +task_id | +setup_id | +flow_id | +flow_name | +data_id | +data_name | +function | +upload_time | +uploader | +uploader_name | +value | +values | +array_data | +
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 144 | +1849043 | +59 | +29015 | +5500 | +sklearn.ensemble.forest.RandomForestClassifier... | +61 | +iris | +predictive_accuracy | +2017-03-03 17:10:12 | +1 | +janvanrijn@gmail.com | +0.946667 | +None | +None | +
| 145 | +1853409 | +59 | +30950 | +5873 | +sklearn.pipeline.Pipeline(Imputer=openml.utils... | +61 | +iris | +predictive_accuracy | +2017-03-21 22:08:01 | +1 | +janvanrijn@gmail.com | +0.960000 | +None | +None | +
| 146 | +6130126 | +59 | +4163633 | +7108 | +sklearn.model_selection._search.RandomizedSear... | +61 | +iris | +predictive_accuracy | +2017-08-21 11:07:40 | +1 | +janvanrijn@gmail.com | +0.960000 | +None | +None | +
| 147 | +6130128 | +59 | +4163634 | +7108 | +sklearn.model_selection._search.RandomizedSear... | +61 | +iris | +predictive_accuracy | +2017-08-21 11:08:06 | +1 | +janvanrijn@gmail.com | +0.946667 | +None | +None | +
| 190 | +6946499 | +59 | +4978397 | +7109 | +sklearn.pipeline.Pipeline(imputation=openmlstu... | +61 | +iris | +predictive_accuracy | +2017-09-02 22:06:32 | +1 | +janvanrijn@gmail.com | +0.920000 | +None | +None | +
Since we are an ambitious bunch of ML practitioners who settle for nothing but the best, and also since most results will not be considered worth the effort if not matching or beating state-of-the-art, we shall aim for the best scores. We’ll sort the filtered results we obtained based on the score or ‘value’ and then extract the components from that run - task and flow.
+task_rf.sort_values(by='value', ascending=False).head()| + | run_id | +task_id | +setup_id | +flow_id | +flow_name | +data_id | +data_name | +function | +upload_time | +uploader | +uploader_name | +value | +values | +array_data | +
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 3549 | +523926 | +59 | +3526 | +2629 | +sklearn.ensemble.forest.RandomForestClassifier(8) | +61 | +iris | +predictive_accuracy | +2016-02-11 22:05:23 | +869 | +p.gijsbers@student.tue.nl | +0.966667 | +None | +None | +
| 4353 | +8955370 | +59 | +6890988 | +7257 | +sklearn.ensemble.forest.RandomForestClassifier... | +61 | +iris | +predictive_accuracy | +2018-04-06 16:32:22 | +3964 | +clear.tsai@gmail.com | +0.960000 | +None | +None | +
| 3587 | +1852682 | +59 | +29263 | +5500 | +sklearn.ensemble.forest.RandomForestClassifier... | +61 | +iris | +predictive_accuracy | +2017-03-15 22:55:18 | +1022 | +rso@randalolson.com | +0.960000 | +None | +None | +
| 4375 | +8886608 | +59 | +6835139 | +7961 | +sklearn.pipeline.Pipeline(Imputer=sklearn.prep... | +61 | +iris | +predictive_accuracy | +2018-03-17 16:46:27 | +5032 | +rashmi.kamath01@gmail.com | +0.960000 | +None | +None | +
| 3107 | +1843272 | +59 | +24071 | +4830 | +sklearn.ensemble.forest.RandomForestClassifier... | +61 | +iris | +predictive_accuracy | +2016-12-08 20:10:03 | +2 | +joaquin.vanschoren@gmail.com | +0.960000 | +None | +None | +
# Fetching the Random Forest flow with the best score
+f = openml.flows.get_flow(2629)
+fOpenML Flow
+===========
+Flow ID.........: 2629 (version 8)
+Flow URL........: https://www.openml.org/f/2629
+Flow Name.......: sklearn.ensemble.forest.RandomForestClassifier
+Flow Description: Flow generated by openml_run
+Upload Date.....: 2016-02-11 21:17:08
+Dependencies....: None# Fetching the run with the best score for
+# Random Forest on Iris
+r = openml.runs.get_run(523926)
+rOpenML Run
+==========
+Uploader Name...: Pieter Gijsbers
+Uploader Profile: https://www.openml.org/u/869
+Metric..........: predictive_accuracy
+Result..........: 0.966667
+Run ID..........: 523926
+Run URL.........: https://www.openml.org/r/523926
+Task ID.........: 59
+Task Type.......: Supervised Classification
+Task URL........: https://www.openml.org/t/59
+Flow ID.........: 2629
+Flow Name.......: sklearn.ensemble.forest.RandomForestClassifier(8)
+Flow URL........: https://www.openml.org/f/2629
+Setup ID........: 3526
+Setup String....: None
+Dataset ID......: 61
+Dataset URL.....: https://www.openml.org/d/61Okay, let’s take a pause and re-assess. From multiple users across the globe, who had uploaded runs to OpenML, for a Random Forest run on the Iris, the best score seen till now is 96.67%. That is certainly better than the naive model we built at the beginning to achieve 95.33%. We had used a basic 10-fold cross-validation to evaluate a Random Forest of 10 trees with a max depth of 2. Let us see, what the best run uses and if it differs from our approach.
+# The scoring metric used
+t.evaluation_measure'predictive_accuracy'# The methodology used for estimations
+t.estimation_procedure{'type': 'crossvalidation',
+ 'parameters': {'number_repeats': '1',
+ 'number_folds': '10',
+ 'percentage': '',
+ 'stratified_sampling': 'true'},
+ 'data_splits_url': 'https://www.openml.org/api_splits/get/59/Task_59_splits.arff'}# The model used
+f.name'sklearn.ensemble.forest.RandomForestClassifier'# The model parameters
+for param in r.parameter_settings:
+ name, value = param['oml:name'], param['oml:value']
+ print("{:<25} : {:<10}".format(name, value))warm_start : False
+oob_score : False
+n_jobs : 1
+verbose : 0
+max_leaf_nodes : None
+bootstrap : True
+min_samples_leaf : 1
+n_estimators : 10
+min_samples_split : 2
+min_weight_fraction_leaf : 0.0
+criterion : gini
+random_state : None
+max_features : auto
+max_depth : None
+class_weight : None As evident, our initial approach is different on two fronts. We didn’t explicitly use stratified sampling for our cross-validation. While the Random Forest hyperparameters are slightly different too (max_depth=None). That definitely sounds like a to-do, however, there is no reason why we should restrict ourselves to Random Forests. Remember, we are aiming big here. Given the number of OpenML users, there must be somebody who got a better score on Iris with some other model. Let us then retrieve that information. Programmatically, of course.
+In summary, we are now going to sort the performance of all scikit-learn based models on Iris dataset as per the task definition with task_id=59.
+# Fetching top performances
+task_df.sort_values(by='value', ascending=False).head()| + | run_id | +task_id | +setup_id | +flow_id | +flow_name | +data_id | +data_name | +function | +upload_time | +uploader | +uploader_name | +value | +values | +array_data | +
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 3630 | +2039748 | +59 | +180922 | +6048 | +sklearn.pipeline.Pipeline(dualimputer=helper.d... | +61 | +iris | +predictive_accuracy | +2017-04-09 01:09:01 | +1104 | +jmapvhoof@gmail.com | +0.986667 | +None | +None | +
| 3631 | +2039750 | +59 | +180924 | +6048 | +sklearn.pipeline.Pipeline(dualimputer=helper.d... | +61 | +iris | +predictive_accuracy | +2017-04-09 01:17:39 | +1104 | +jmapvhoof@gmail.com | +0.986667 | +None | +None | +
| 3624 | +2012939 | +59 | +157622 | +6048 | +sklearn.pipeline.Pipeline(dualimputer=helper.d... | +61 | +iris | +predictive_accuracy | +2017-04-06 23:29:28 | +1104 | +jmapvhoof@gmail.com | +0.986667 | +None | +None | +
| 3618 | +2012930 | +59 | +157613 | +6048 | +sklearn.pipeline.Pipeline(dualimputer=helper.d... | +61 | +iris | +predictive_accuracy | +2017-04-06 23:00:24 | +1104 | +jmapvhoof@gmail.com | +0.986667 | +None | +None | +
| 3626 | +2012941 | +59 | +157624 | +6048 | +sklearn.pipeline.Pipeline(dualimputer=helper.d... | +61 | +iris | +predictive_accuracy | +2017-04-07 01:36:00 | +1104 | +jmapvhoof@gmail.com | +0.986667 | +None | +None | +
# Fetching best performing flow
+f = openml.flows.get_flow(6048)
+fOpenML Flow
+===========
+Flow ID.........: 6048 (version 1)
+Flow URL........: https://www.openml.org/f/6048
+Flow Name.......: sklearn.pipeline.Pipeline(dualimputer=helper.dual_imputer.DualImputer,nusvc=sklearn.svm.classes.NuSVC)
+Flow Description: Automatically created scikit-learn flow.
+Upload Date.....: 2017-04-06 22:42:59
+Dependencies....: sklearn==0.18.1
+numpy>=1.6.1
+scipy>=0.9# Fetching best performing run
+r = openml.runs.get_run(2012943)
+
+# The model parameters
+for param in r.parameter_settings:
+ name, value = param['oml:name'], param['oml:value']
+ print("{:<25} : {:<10}".format(name, value))steps : [('DualImputer', <helper.dual_imputer.DualImputer object at 0x7ff618e4d908>), ('nusvc', NuSVC(cache_size=200, class_weight=None, coef0=0.0,
+ decision_function_shape=None, degree=3, gamma='auto', kernel='linear',
+ max_iter=-1, nu=0.3, probability=True, random_state=3, shrinking=True,
+ tol=3.2419092644286417e-05, verbose=False))]
+cache_size : 200
+class_weight : None
+coef0 : 0.0
+decision_function_shape : None
+degree : 3
+gamma : auto
+kernel : linear
+max_iter : -1
+nu : 0.3
+probability : True
+random_state : 3
+shrinking : True
+tol : 3.24190926443e-05
+verbose : False The highest score obtained among the uploaded results is 98.67% using a variant of SVM. However, if we check the corresponding flow description, we see that it is using an old scikit-learn version (0.18.1) and therefore may not be possible to replicate the exact results. However, in order to improve from our score of 95.33%, we should try running a nu-SVC on the same problem and see where we stand. Let’s go for it. Via OpenML, of course.
+import openml
+import numpy as np
+from sklearn.svm import NuSVC# Building the NuSVC model object with parameters found
+clf = NuSVC(cache_size=200, class_weight=None, coef0=0.0,
+ decision_function_shape=None, degree=3, gamma='auto', kernel='linear',
+ max_iter=-1, nu=0.3, probability=True, random_state=3, shrinking=True,
+ tol=3.2419092644286417e-05, verbose=False)# Obtaining task used earlier
+t = openml.tasks.get_task(59)
+tOpenML Classification Task
+==========================
+Task Type Description: https://www.openml.org/tt/1
+Task ID..............: 59
+Task URL.............: https://www.openml.org/t/59
+Estimation Procedure.: crossvalidation
+Evaluation Measure...: predictive_accuracy
+Target Feature.......: class
+# of Classes.........: 3
+Cost Matrix..........: Available# Running the model on the task
+# Internally, the model will be made into
+# an OpenML flow and we can choose to retrieve it
+r, f = openml.runs.run_model_on_task(model=clf, task=t, upload_flow=False, return_flow=True)
+fOpenML Flow
+===========
+Flow Name.......: sklearn.svm.classes.NuSVC
+Flow Description: Nu-Support Vector Classification.
+
+Similar to SVC but uses a parameter to control the number of support
+vectors.
+
+The implementation is based on libsvm.
+Dependencies....: sklearn==0.21.3
+numpy>=1.6.1
+scipy>=0.9# To obtain the score (without uploading)
+## r.publish() can be used to upload these results
+## need to sign-in to https://www.openml.org/
+score = []
+evaluations = r.fold_evaluations['predictive_accuracy'][0]
+for key in evaluations:
+ score.append(evaluations[key])
+print(np.mean(score))0.9866666666666667Lo and behold! We hit the magic number. I personally would have never tried out NuSVC and would have stuck around tweaking hyperparameters of the Random Forest. This is a new discovery of sorts for sure. I wonder though if anybody has tried XGBoost on Iris?
+In any case, we can now upload the results of this run to OpenML using:
+r.publish()OpenML Run
+==========
+Uploader Name: None
+Metric.......: None
+Run ID.......: 10464835
+Run URL......: https://www.openml.org/r/10464835
+Task ID......: 59
+Task Type....: None
+Task URL.....: https://www.openml.org/t/59
+Flow ID......: 18579
+Flow Name....: sklearn.svm.classes.NuSVC
+Flow URL.....: https://www.openml.org/f/18579
+Setup ID.....: None
+Setup String.: Python_3.6.9. Sklearn_0.21.3. NumPy_1.16.4. SciPy_1.4.1. NuSVC(cache_size=200, class_weight=None, coef0=0.0,
+ decision_function_shape=None, degree=3, gamma='auto', kernel='linear',
+ max_iter=-1, nu=0.3, probability=True, random_state=3, shrinking=True,
+ tol=3.241909264428642e-05, verbose=False)
+Dataset ID...: 61
+Dataset URL..: https://www.openml.org/d/61One would need to sign-in to https://www.openml.org/ and generate their respective apikey. The results would then be available for everyone to view and who knows, you can have your name against the best-ever performance measured on the Iris dataset!
+This post was in no ways intended to be a be-all-end-all guide to OpenML. The primary goal was to help form an acquaintance with the OpenML terminologies, introduce the API, establish connections with the general ML practices, and give a sneak-peek into the potential benefits of working together as a community. For a better understanding of OpenML, please explore the documentation. If one desires to continue from the examples given in this post and explore further, kindly refer to the API.
+OpenML-Python is an open-source project and contributions from everyone in the form of Issues and Pull Requests are most welcome. Contribution to the OpenML community is in fact not limited to code contribution. Every single user can make the community richer by sharing data, experiments, results, using OpenML.
+As ML practitioners, we may be dependent on tools for our tasks. However, as a collective, we can juice out its potential to a larger extent. Let us together, make ML more transparent, more democratic!
+Special thanks to Heidi, Bilge, Sahithya, Matthias, Ashwin for the ideas, feedback, and support.
+Related readings: * To get started with OpenML-Python * OpenML-Python Github * The OpenML website * Miscellaneous reading on OpenML * To get in touch!
+ + +Deep learning is facing a reproducibility crisis right now[1]. The scale of experiments and there are numerous hyperparameters that affect performance, which makes it hard for the author to write a reproducibility document. The current best way to make an experiment reproducible is to upload the code. However, that’s not optimal in a lot of situations where we have a huge undocumented codebase and someone would like to just reproduce the model. OpenML[2] is an online machine learning platform for sharing and organizing data, machine learning algorithms and experiments. Until now we only provided support for classical machine learning and libraries like Sklearn and MLR. We see there is a huge need for reproducible deep learning now. To solve this issue OpenML is launching its deep learning plugins for popular deep learning libraries like Keras, MXNet, and Pytorch.
+Here we have a small tutorial on how to use our pytorch extension with MNIST dataset.
+Setup
To install openml and openml pytorch extension execute this instruction in your terminal
pip install openml openml_pytorch
!pip install openml openml_pytorchCollecting openml
+ Downloading https://files.pythonhosted.org/packages/68/5b/cd32bb85651eccebfb489cc6ef7f060ce0f62350a6239127e398313090cc/openml-0.10.2.tar.gz (158kB)
+
+ |██ | 10kB 28.5MB/s eta 0:00:01
+ |████▏ | 20kB 6.1MB/s eta 0:00:01
+ |██████▏ | 30kB 8.6MB/s eta 0:00:01
+ |████████▎ | 40kB 10.9MB/s eta 0:00:01
+ |██████████▎ | 51kB 7.2MB/s eta 0:00:01
+ |████████████▍ | 61kB 8.4MB/s eta 0:00:01
+ |██████████████▍ | 71kB 9.6MB/s eta 0:00:01
+ |████████████████▌ | 81kB 10.7MB/s eta 0:00:01
+ |██████████████████▌ | 92kB 8.5MB/s eta 0:00:01
+ |████████████████████▋ | 102kB 9.3MB/s eta 0:00:01
+ |██████████████████████▊ | 112kB 9.3MB/s eta 0:00:01
+ |████████████████████████▊ | 122kB 9.3MB/s eta 0:00:01
+ |██████████████████████████▉ | 133kB 9.3MB/s eta 0:00:01
+ |████████████████████████████▉ | 143kB 9.3MB/s eta 0:00:01
+ |███████████████████████████████ | 153kB 9.3MB/s eta 0:00:01
+ |████████████████████████████████| 163kB 9.3MB/s
+Collecting openml_pytorch
+ Downloading https://files.pythonhosted.org/packages/5b/a4/8c69a041e7929d93460db17cf276abfb7b49af9c3d5077bee1c52101ba4c/openml_pytorch-0.0.1-py3-none-any.whl
+Collecting liac-arff>=2.4.0
+ Downloading https://files.pythonhosted.org/packages/e9/35/fbc9217cfa91d98888b43e1a19c03a50d716108c58494c558c65e308f372/liac-arff-2.4.0.tar.gz
+Collecting xmltodict
+ Downloading https://files.pythonhosted.org/packages/28/fd/30d5c1d3ac29ce229f6bdc40bbc20b28f716e8b363140c26eff19122d8a5/xmltodict-0.12.0-py2.py3-none-any.whl
+Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from openml) (2.21.0)
+Requirement already satisfied: scikit-learn>=0.18 in /usr/local/lib/python3.6/dist-packages (from openml) (0.22.2.post1)
+Requirement already satisfied: python-dateutil in /usr/local/lib/python3.6/dist-packages (from openml) (2.8.1)
+Requirement already satisfied: pandas>=0.19.2 in /usr/local/lib/python3.6/dist-packages (from openml) (1.0.3)
+Requirement already satisfied: scipy>=0.13.3 in /usr/local/lib/python3.6/dist-packages (from openml) (1.4.1)
+Requirement already satisfied: numpy>=1.6.2 in /usr/local/lib/python3.6/dist-packages (from openml) (1.18.2)
+Collecting torch==1.2.0
+ Downloading https://files.pythonhosted.org/packages/30/57/d5cceb0799c06733eefce80c395459f28970ebb9e896846ce96ab579a3f1/torch-1.2.0-cp36-cp36m-manylinux1_x86_64.whl (748.8MB)
+ |████████████████████████████████| 748.9MB 14kB/s
+Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->openml) (3.0.4)
+Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->openml) (1.24.3)
+Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->openml) (2020.4.5.1)
+Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->openml) (2.8)
+Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.6/dist-packages (from scikit-learn>=0.18->openml) (0.14.1)
+Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.6/dist-packages (from python-dateutil->openml) (1.12.0)
+Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas>=0.19.2->openml) (2018.9)
+Building wheels for collected packages: openml, liac-arff
+ Building wheel for openml (setup.py) ... done
+ Created wheel for openml: filename=openml-0.10.2-cp36-none-any.whl size=190318 sha256=50c2fd823d13904f246bf30997a2464379c393377cfd77f74b5dace4935db99c
+ Stored in directory: /root/.cache/pip/wheels/71/ec/5f/aaad9e184680b0b8f1a02ff0ec640cace5adf5bff7bb0af1b4
+ Building wheel for liac-arff (setup.py) ... done
+ Created wheel for liac-arff: filename=liac_arff-2.4.0-cp36-none-any.whl size=13335 sha256=fca5bc5e07e3fe4f591cbe79968a42e43f535b3bd3e4be77b62c901d46feaaa9
+ Stored in directory: /root/.cache/pip/wheels/d1/6a/e7/529dc54d76ecede4346164a09ae3168df358945612710f5203
+Successfully built openml liac-arff
+ERROR: torchvision 0.5.0 has requirement torch==1.4.0, but you'll have torch 1.2.0 which is incompatible.
+Installing collected packages: liac-arff, xmltodict, openml, torch, openml-pytorch
+ Found existing installation: torch 1.4.0
+ Uninstalling torch-1.4.0:
+ Successfully uninstalled torch-1.4.0
+Successfully installed liac-arff-2.4.0 openml-0.10.2 openml-pytorch-0.0.1 torch-1.2.0 xmltodict-0.12.0Let’s import the necessary libraries
+import torch.nn
+import torch.optim
+import openml
+import openml_pytorch
+
+import loggingSet the apikey for openml python library, you can find your api key in your openml.org account
+openml.config.apikey = 'key'Define a sequential network that does initial image reshaping and normalization model
+processing_net = torch.nn.Sequential(
+ openml_pytorch.layers.Functional(function=torch.Tensor.reshape,
+ shape=(-1, 1, 28, 28)),
+ torch.nn.BatchNorm2d(num_features=1)
+)
+print(processing_net)Sequential(
+ (0): Functional()
+ (1): BatchNorm2d(1, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
+)Define a sequential network that does the extracts the features from the image.
+features_net = torch.nn.Sequential(
+ torch.nn.Conv2d(in_channels=1, out_channels=32, kernel_size=5),
+ torch.nn.LeakyReLU(),
+ torch.nn.MaxPool2d(kernel_size=2),
+ torch.nn.Conv2d(in_channels=32, out_channels=64, kernel_size=5),
+ torch.nn.LeakyReLU(),
+ torch.nn.MaxPool2d(kernel_size=2),
+)
+print(features_net)Sequential(
+ (0): Conv2d(1, 32, kernel_size=(5, 5), stride=(1, 1))
+ (1): LeakyReLU(negative_slope=0.01)
+ (2): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
+ (3): Conv2d(32, 64, kernel_size=(5, 5), stride=(1, 1))
+ (4): LeakyReLU(negative_slope=0.01)
+ (5): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
+)Define a sequential network that flattens the features and compiles the results into probabilities for each digit.
+results_net = torch.nn.Sequential(
+ openml_pytorch.layers.Functional(function=torch.Tensor.reshape,
+ shape=(-1, 4 * 4 * 64)),
+ torch.nn.Linear(in_features=4 * 4 * 64, out_features=256),
+ torch.nn.LeakyReLU(),
+ torch.nn.Dropout(),
+ torch.nn.Linear(in_features=256, out_features=10),
+)
+print(results_net)Sequential(
+ (0): Functional()
+ (1): Linear(in_features=1024, out_features=256, bias=True)
+ (2): LeakyReLU(negative_slope=0.01)
+ (3): Dropout(p=0.5, inplace=False)
+ (4): Linear(in_features=256, out_features=10, bias=True)
+)The main network, composed of the above specified networks.
+model = torch.nn.Sequential(
+ processing_net,
+ features_net,
+ results_net
+)
+print(model)Sequential(
+ (0): Sequential(
+ (0): Functional()
+ (1): BatchNorm2d(1, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
+ )
+ (1): Sequential(
+ (0): Conv2d(1, 32, kernel_size=(5, 5), stride=(1, 1))
+ (1): LeakyReLU(negative_slope=0.01)
+ (2): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
+ (3): Conv2d(32, 64, kernel_size=(5, 5), stride=(1, 1))
+ (4): LeakyReLU(negative_slope=0.01)
+ (5): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
+ )
+ (2): Sequential(
+ (0): Functional()
+ (1): Linear(in_features=1024, out_features=256, bias=True)
+ (2): LeakyReLU(negative_slope=0.01)
+ (3): Dropout(p=0.5, inplace=False)
+ (4): Linear(in_features=256, out_features=10, bias=True)
+ )
+)Download the OpenML task for the mnist 784 dataset.
+task = openml.tasks.get_task(3573)Run the model on the task and publish the results on openml.org
+
+run = openml.runs.run_model_on_task(model, task, avoid_duplicate_runs=False)
+
+run.publish()
+
+print('URL for run: %s/run/%d' % (openml.config.server, run.run_id))URL for run: https://www.openml.org/api/v1/xml/run/10452577By going to the published URL you can check the model performance and other metadata
+
We hope that openml deep learning plugins can help in reproducing deep learning experiments and provide a universal reproducibility platform for the experiments. Here are the links of all supported deep learning plugins right now:
+MXNet: https://github.com/openml/openml-mxnet
Keras: https://github.com/openml/openml-keras
Pytorch: https://github.com/openml/openml-pytorch
ONNX: https://github.com/openml/openml-onnx
There are examples of how to use these libraries in the Github repos. These libraries are in the development stage right now so we would appreciate any feedback on Github issues of these libraries. Links:
+Over the past few months at OpenML, we have been experimenting with LLM models in an attempt to improve the search experience for our users. While our existing implementation uses ElasticSearch, we wanted to also have the option of having a more “semantic” search experience.
+Aside from the usual RAG pipeline that everyone and their grandparents seems to be using these days, we also wanted to experiment with using an LLM to semi-automatically generate filters for our search queries. While it may not seem like a big feature, it is something that has always been a bit of an annoyance for some of our users.
+So what does this entail? Consider the interface we have at the moment. We have a search bar at the top, and subsequently a bunch of filters that users can use to narrow down their search. While this works pretty well as is, how about trying to automate it a bit.
+In summary, we want a query like “find me a large dataset with multiple classes of flowers” to automatically generate filters like “classification”, “multiclass”, “sort by size of dataset” etc.
+
Think about the first time you used ChatGPT. What stood out to you? Was it how well it could elaborate on a topic? Or was it how creative it could be? The temperature parameter in LLMs is what controls this.
+How can we control creativity? Well, saying that we can directly control creativity is a bit of a stretch. We can however use a workaround.
+Do you remember the softmax function? The function that takes a vector of arbitrary real-valued scores and squashes it into a vector of probabilities that sum to 1. The inputs to the softmax function are the unnormalized log likelikhoods or the raw per class score assigned by the model.
+The softmax function is defined as: \[ \text{softmax}(x_i) = \frac{e^{x_i}}{\sum_{j=1}^{k} e^{x_j}} \]
+If we want more control over the distribution of the probabilities, we can use a temperature parameter. This would look like: \[ \text{softmax}(x_i) = \frac{e^{x_i/T}}{\sum_{j=1}^{k} e^{x_j/T}} \] where \(T\) is the temperature parameter.
+If \(T = 1\), the softmax function is the same as the original softmax function.
If \(T > 1\), the probabilities will become “flatter”. Since the difference between the probabilities will be less, the model can be more exploratory aka more creative.
If \(T < 1\), the distribution of the probabilities are “peakier”. There will be a higher difference between the probabilities, leading to the model being more confident in its predictions, but also less creative.
from tqdm import tqdm
+import regex as re
+# LangChain supports many other chat models. Here, we're using Ollama
+from langchain_community.chat_models import ChatOllama
+from langchain_core.output_parsers import StrOutputParser
+from langchain_core.prompts import ChatPromptTemplate
+from typing import List, Dict, Any
+import numpy as np
+import seaborn as sns
+import matplotlib.pyplot as plt
+import pandas as pd
+sns.set_theme(style="white")def softmax(input, t=1.0):
+ ex = np.exp(input/t)
+ sum = np.sum(ex, axis=0)
+ return ex / sum# plot softmax over a range of inputs
+x = np.arange(0,1.0, 0.01)
+t = np.array([0.1,.5, .8, 1.0])
+y = np.array([softmax(x, ti) for ti in t])
+
+# Create a DataFrame for Seaborn
+data = pd.DataFrame({
+ 'x': np.tile(x, len(t)),
+ 'softmax': np.concatenate(y),
+ 't': np.repeat(t, len(x))
+})
+
+# Plotting with Seaborn
+plt.figure(figsize=(8, 6))
+sns.lineplot(data=data, x='x', y='softmax', hue='t', palette='viridis')
+plt.xlabel('x')
+plt.ylabel('softmax(x)')
+plt.title('Softmax Function for Different Values of t')
+plt.legend(title='t')
+plt.show()
Now, we can focus on testing the effects of temperature for our use case. We are using the llama3 model for our experiments. The experiments are being run on a 2023 MacBook Pro with an M3 chip and 18GB memory.
We need to first think of a prompt that we can use for our experiments. This prompt can be thought of as an instruction that the model uses along with the query to generate answers. To make it easier for us to use, we only want one/two word answers and for now we are only focusing on a small subset of the filters that we want our model to understand.
+prompt = """User Query : {query}
+Based on the query, answer the following questions one by one in one or two words only and a maximum of two with commas only if asked for. Use only the information given and do not make up answers -
+Does the user care about the size of the dataset? Yes/No and if yes, ascending/descending.
+Does the user want to sort by number of downloads? Yes/No.
+Does the user care about missing values? Yes/No.
+If it seems like the user wants a classification dataset, is it binary/multi-class/multi-label? If not, say none.
+"""query = "Find me a big classification dataset about mushrooms"Since we are using the langchain and ollama libraries for our experiments, we follow their API and create a chain. The template uses string formatting to insert the prompt and the query into the chain.
def create_chain(prompt , temperature, llm_model = "llama3"):
+ prompt = ChatPromptTemplate.from_template(prompt)
+ llm = ChatOllama(model=llm_model, temperature=temperature)
+ chain = prompt | llm | StrOutputParser()
+ return chainTo make it easier for us to analyze the results, we generate an example answer and then see see if any further processing is needed.
+# functiont to parse responses like this to a list of yes/no/none/yes,aescending/no etc
+def parse_response(response):
+ # split by new line and remove first two lines (here are the answers:)
+ response = response.split('\n')[2::]
+ # if response has a question mark, split by question mark and remove empty strings
+ for i in range(len(response)):
+ if '?' in response[i]:
+ response[i] = response[i].split('?')[1].strip()
+ # replace full stops with empty strings
+ response = [x.replace('.','') for x in response]
+ response = [x for x in response if x]
+ return responsechain = create_chain(prompt, 0.5)
+response = chain.invoke({"query": query})
+print(response)Here are the answers:
+
+1. Does the user care about the size of the dataset?
+Yes, ascending.
+
+2. Does the user want to sort by number of downloads?
+No
+
+3. Does the user care about missing values?
+No
+
+4. Is it a classification dataset? If so, is it binary/multi-class/multi-label?
+Yes, multi-classYay, it works. We now write a function to generate results for different temperatures.
+def generate_results_for_temp(query:str, range_of_temps : np.ndarray) -> List[List[str]]:
+ results = []
+ for temperature in tqdm(range_of_temps):
+ chain = create_chain(prompt, temperature)
+ response = chain.invoke({"query": query})
+ results.append(parse_response(response))
+ return results
+ It is time to run the experiments and plot the results. We write a function to plot the results in a stripplot to see the distribution of the answers for different temperatures.
def plot_yes_no(df: pd.DataFrame, title:str) -> None:
+ fig, axs = plt.subplots(2, 2, figsize=(15, 15))
+ fig.suptitle(title)
+ sns.stripplot(data=df, x='size', y='temperature', ax=axs[0, 0], hue='size')
+ sns.stripplot(data=df, x='sort_by_downloads', y='temperature', ax=axs[0, 1], hue='sort_by_downloads')
+ sns.stripplot(data=df, x='missing_values', y='temperature', ax=axs[1, 0], hue='missing_values')
+ sns.stripplot(data=df, x='classification_type', y='temperature', ax=axs[1, 1], hue='classification_type')
+ # tilt x axis labels
+ for ax in axs.flat:
+ ax.set_xticklabels(ax.get_xticklabels(), rotation=30, horizontalalignment='right')
+ plt.show()Sometimes, the model returns an extra field, we combine the last two fields to plot the results. (This is a bit of a hack, but it works for now and is ONLY used for plotting)
+def combine_last_two_elements(lst):
+ # Check if the list has at least two elements
+ if len(lst) > 4:
+ # Combine the last two elements with a space separator
+ combined_element = lst[-2] + ' ' + lst[-1]
+
+ # Create a new list with combined element instead of the last two
+ return lst[:-2] + [combined_element]
+ else:
+ return lstOut first experiment is a rather simple query, “Find me a big classification dataset about mushrooms”. As you can probably guess, we are looking for a dataset that is large, is a classification dataset and is about mushrooms.
+range_of_temps = np.linspace(0, 1, 20)
+query = "Find me a big classification dataset about mushrooms"
+results1 = generate_results_for_temp(query, range_of_temps)100%|██████████| 20/20 [00:49<00:00, 2.49s/it]results1 = [[y for y in x if all(sub not in y for sub in ["If", ":"])] for x in results1]df = pd.DataFrame(results1, columns = ['size', 'sort_by_downloads', 'missing_values', 'classification_type'])
+df['temperature'] = range_of_temps
+plot_yes_no(df, title = query)/var/folders/_f/ng_zp8zj2dgf828sb6s5wdb00000gn/T/ipykernel_42184/3521201435.py:10: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
+ ax.set_xticklabels(ax.get_xticklabels(), rotation=30, horizontalalignment='right')
+/var/folders/_f/ng_zp8zj2dgf828sb6s5wdb00000gn/T/ipykernel_42184/3521201435.py:10: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
+ ax.set_xticklabels(ax.get_xticklabels(), rotation=30, horizontalalignment='right')
+/var/folders/_f/ng_zp8zj2dgf828sb6s5wdb00000gn/T/ipykernel_42184/3521201435.py:10: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
+ ax.set_xticklabels(ax.get_xticklabels(), rotation=30, horizontalalignment='right')
+/var/folders/_f/ng_zp8zj2dgf828sb6s5wdb00000gn/T/ipykernel_42184/3521201435.py:10: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
+ ax.set_xticklabels(ax.get_xticklabels(), rotation=30, horizontalalignment='right')
Rather interesting don’t you think? At higher temperatures, the model gets the answers wrong. Even at a temperature slightly above 0.1, the model starts adding extra information to it’s answers.
+Did you notice that I tried to remove sentences that started with “If”? There are more examples of this later, but this is because at higher temperatures, the model tends to add random sentences to the answers and this makes it quite hard to plot them.
+Our second experiment is super easy. “Find me a dataset that has a lot of missing values and order by number of downloads”. As you can obviously guess, we are looking for a dataset that has a lot of missing values and we want to order the results by the number of downloads.
+range_of_temps = np.linspace(0, 1, 20)
+query = "Find me a dataset that has a lot of missing values and order by number of downloads"
+results2 = generate_results_for_temp(query, range_of_temps)
+results2 = [[y for y in x if "so" not in y] for x in results2]100%|██████████| 20/20 [00:34<00:00, 1.74s/it]df = pd.DataFrame(results2, columns = ['size', 'sort_by_downloads', 'missing_values', 'classification_type'])
+df['temperature'] = range_of_temps
+plot_yes_no(df, title = query)/var/folders/_f/ng_zp8zj2dgf828sb6s5wdb00000gn/T/ipykernel_42184/3521201435.py:10: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
+ ax.set_xticklabels(ax.get_xticklabels(), rotation=30, horizontalalignment='right')
+/var/folders/_f/ng_zp8zj2dgf828sb6s5wdb00000gn/T/ipykernel_42184/3521201435.py:10: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
+ ax.set_xticklabels(ax.get_xticklabels(), rotation=30, horizontalalignment='right')
+/var/folders/_f/ng_zp8zj2dgf828sb6s5wdb00000gn/T/ipykernel_42184/3521201435.py:10: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
+ ax.set_xticklabels(ax.get_xticklabels(), rotation=30, horizontalalignment='right')
+/var/folders/_f/ng_zp8zj2dgf828sb6s5wdb00000gn/T/ipykernel_42184/3521201435.py:10: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
+ ax.set_xticklabels(ax.get_xticklabels(), rotation=30, horizontalalignment='right')
Hmm, same as before. The model starts adding extra information at higher temperatures and starts getting the answers wrong. (Yes, No?? ) What kind of answer is that?
+range_of_temps = np.linspace(0, 1, 20)
+query = "Find me a dataset that has 10 classes and sort by number of downloads"
+results3 = generate_results_for_temp(query, range_of_temps)100%|██████████| 20/20 [00:55<00:00, 2.80s/it]results3 = [combine_last_two_elements(x) for x in results3]df = pd.DataFrame(results3, columns = ['size', 'sort_by_downloads', 'missing_values', 'classification_type'])
+df['temperature'] = range_of_temps
+plot_yes_no(df, title = query)/var/folders/_f/ng_zp8zj2dgf828sb6s5wdb00000gn/T/ipykernel_42184/3521201435.py:10: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
+ ax.set_xticklabels(ax.get_xticklabels(), rotation=30, horizontalalignment='right')
+/var/folders/_f/ng_zp8zj2dgf828sb6s5wdb00000gn/T/ipykernel_42184/3521201435.py:10: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
+ ax.set_xticklabels(ax.get_xticklabels(), rotation=30, horizontalalignment='right')
+/var/folders/_f/ng_zp8zj2dgf828sb6s5wdb00000gn/T/ipykernel_42184/3521201435.py:10: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
+ ax.set_xticklabels(ax.get_xticklabels(), rotation=30, horizontalalignment='right')
+/var/folders/_f/ng_zp8zj2dgf828sb6s5wdb00000gn/T/ipykernel_42184/3521201435.py:10: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
+ ax.set_xticklabels(ax.get_xticklabels(), rotation=30, horizontalalignment='right')
This seems to have been very easy for the model. But as always, the model starts adding extra information at higher temperatures. A lot of extra information in fact. Even though the prompt says to ONLY answer with one or two words
+range_of_temps = np.linspace(0, 1, 20)
+query = "Find me a dataset that 2 classes and is a big dataset"
+results4 = generate_results_for_temp(query, range_of_temps)100%|██████████| 20/20 [00:42<00:00, 2.14s/it]results4 = [combine_last_two_elements(x) for x in results4]df = pd.DataFrame(results4, columns = ['size', 'sort_by_downloads', 'missing_values', 'classification_type'])
+df['temperature'] = range_of_temps
+plot_yes_no(df, title = query)/var/folders/_f/ng_zp8zj2dgf828sb6s5wdb00000gn/T/ipykernel_42184/3521201435.py:10: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
+ ax.set_xticklabels(ax.get_xticklabels(), rotation=30, horizontalalignment='right')
+/var/folders/_f/ng_zp8zj2dgf828sb6s5wdb00000gn/T/ipykernel_42184/3521201435.py:10: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
+ ax.set_xticklabels(ax.get_xticklabels(), rotation=30, horizontalalignment='right')
+/var/folders/_f/ng_zp8zj2dgf828sb6s5wdb00000gn/T/ipykernel_42184/3521201435.py:10: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
+ ax.set_xticklabels(ax.get_xticklabels(), rotation=30, horizontalalignment='right')
+/var/folders/_f/ng_zp8zj2dgf828sb6s5wdb00000gn/T/ipykernel_42184/3521201435.py:10: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
+ ax.set_xticklabels(ax.get_xticklabels(), rotation=30, horizontalalignment='right')
Notice how some things changed? At higher temperatures, we get extended answers.
+In conclusion, we can see that we should probably stick to lower temperatures for our use case. As we go higher, the model starts being more “creative” and either adds extra information to the answers or gets them wrong. While this behaviour might be useful in cases like creative writing, it is not something we want in our search.
+Using LLMs can sometimes be a bit of a hit or miss. But of course, learning to control it’s parameters can help us get the most out of it. This blog post was just a simple experiment, but in the deluge of content made by people who have no idea what Softmax is, I hope this was helpful.
+ + +Twice a year the OpenML community organizes an OpenML workshop. Next week we will kick off our fall workshop, which will be hosted virtually and is open to everyone! At the workshop we work on the development and maintenance of the OpenML platform (website, APIs and extensions), but also encourage discussions on a broad range of topics, from using the platform to building the platform to building the community. If you are already sure you want to join, please register and we will see you next week!
+The workshops are the perfect time to get involved with the OpenML community. We welcome first time contributors to the project. You can contribute in many ways, such as giving feedback, cleaning data, or helping with design, planning, or content. If you are looking to contribute to the software projects, we are there to help you get started on beginner issues. Even better, the workshop coincides with Hacktoberfest, which means you can earn a shirt through your open source software contributions! To get an even better idea of what happens at an OpenML workshop, read the write-up of last year’s fall workshop and look at the workshop agenda.
+In this blog post we will go over the basics on how you can participate in the workshop. This includes pointers to our code repositories, the platform we will use (gather.town), and participating in Hacktoberfest.
+Where? openml at gather.town
+When? 26th through 30th of October 2020, 09.00-17.00 CET.
+What? Discussions about OpenML, contributing to any OpenML project, or experimenting with OpenML.
Please register if you plan to join. It is not required to attend the workshop in full. You are free to join only for a day or specific discussions.
+However, it is helpful to indicate which meetings you want to attend, so we know who to expect.
The workshop agenda is public. If you see a meeting which you would like to join, just add your name to the list. Feel free to propose a new meeting by adding a proposal to the meeting list!
+This fall, we will host the OpenML Hackathon at our gather.town space. Gather.town is a web-based platform that allows you to video conference with other people, but with a twist. Rather than the conventional conference call, in gather.town you control an avatar that can navigate a virtual world. You will automatically be connected to anyone that is within the vicinity of your avatar. This means you can more organically join groups of people, or walk from one meeting room to the next. Here is a preview of our OpenML map: 
At a glance you see two types of areas: those marked with purple and those that are not marked. Areas that are not marked (e.g. the main lobby) function exactly as described above. However, whenever you are in a purple area you will automatically be connected to everyone else in the same purple area. So whenever you are interested in joining people discussing the new openml website, you just head over to that room!
+In the workshop agenda you will find various breakout sessions. These cover a wide range of topics, and we will briefly cover a few of them to give you an impression.
+The Dataset Quality break-out session discusses what quality standards we want to uphold for datasets on OpenML. Ideally, we would like all our datasets perfectly annotated with a description of how the data was collected, what each features means, and what the feature types are. In reality, datasets are collected from different sources and often uploaded by someone other than the original creator. This leads to issues such as incomplete data, badly annotated columns, or multiple uploads of the same data.How can we ensure that despite these practical issues, it remains easy to find datasets of interest on OpenML? How can we improve the dataset quality of datasets uploaded to OpenML? How can we set these new standards and guidelines in a maintainable way? How can we automate annotation of attribute types to the dataset? How can we detect corrupt datasets before uploading it to server?
The Website Feedback break-out session is to discuss the newly designed OpenML website. We will appreciate any feedback on frontend, functionalities and limitations of the new openml website.
We recently discussed that OpenML wants to improve support for a wider range of datasets. For this reason we are moving away from the ARFF format. The Dataset Format break-out session discusses this transition. Our current plan is to migrate to parquet format, we will be discussing about limitations and advantages of new format and how can we integrate parquet to OpenML.
We welcome your participation in any these sessions.
+And if you have an interesting topic we should discuss, feel free to submit your own breakout proposal!
Hacktoberfest is a global event to support open source software. If you contribute “enough” to open source projects, you will also be eligible to receive a free t-shirt and some other swag! Our repositories welcome Hacktoberfest participation, so whenever you contribute to our repositories, you are working towards your Hacktoberfest goal. The rules are simple: - Sign up at Hacktoberfest with Github credentials - Find an issue you can help with (more on that below) - Submit a pull request resolving the issue - To be eligible for the t-shirt and swag, you must open four pull requests (this is the “enough” part)
+
You can browse our repositories and identify issues which you would like to work on. To make things easier, here are the list of openml repositories that are extra prepared for Hacktoberfest. These repositories have core developers registered to attend the workshop, and have marked issues which are suitable for newcomers with the hacktoberfest label.
Please be sure to check the respective repositories contribution guidelines, and don’t be afraid to ask questions! We were all new to open source once and are happy to help you get started too. If you are just getting your feet wet with open source contribution, a great way to get started it to just try and use the packages/website. If you see documentation which needs clarification, or perhaps even just a simple typo, we appreciate your help in fixing it!
+If you have any questions about the workshop, feel free to leave a comment.
+ + +${missingFields[0]} field.`,
+ message: `The items being returned for this search do not include all the required fields. Please ensure that your index items include the ${missingFields[0]} field or use index-fields in your _quarto.yml file to specify the field names.`,
+ };
+ } else if (missingFields.length > 1) {
+ const missingFieldList = missingFields
+ .map((field) => {
+ return `${field}`;
+ })
+ .join(", ");
+
+ throw {
+ name: `Error: Search index is missing the following fields: ${missingFieldList}.`,
+ message: `The items being returned for this search do not include all the required fields. Please ensure that your index items includes the following fields: ${missingFieldList}, or use index-fields in your _quarto.yml file to specify the field names.`,
+ };
+ }
+ }
+}
+
+let lastQuery = null;
+function showCopyLink(query, options) {
+ const language = options.language;
+ lastQuery = query;
+ // Insert share icon
+ const inputSuffixEl = window.document.body.querySelector(
+ ".aa-Form .aa-InputWrapperSuffix"
+ );
+
+ if (inputSuffixEl) {
+ let copyButtonEl = window.document.body.querySelector(
+ ".aa-Form .aa-InputWrapperSuffix .aa-CopyButton"
+ );
+
+ if (copyButtonEl === null) {
+ copyButtonEl = window.document.createElement("button");
+ copyButtonEl.setAttribute("class", "aa-CopyButton");
+ copyButtonEl.setAttribute("type", "button");
+ copyButtonEl.setAttribute("title", language["search-copy-link-title"]);
+ copyButtonEl.onmousedown = (e) => {
+ e.preventDefault();
+ e.stopPropagation();
+ };
+
+ const linkIcon = "bi-clipboard";
+ const checkIcon = "bi-check2";
+
+ const shareIconEl = window.document.createElement("i");
+ shareIconEl.setAttribute("class", `bi ${linkIcon}`);
+ copyButtonEl.appendChild(shareIconEl);
+ inputSuffixEl.prepend(copyButtonEl);
+
+ const clipboard = new window.ClipboardJS(".aa-CopyButton", {
+ text: function (_trigger) {
+ const copyUrl = new URL(window.location);
+ copyUrl.searchParams.set(kQueryArg, lastQuery);
+ copyUrl.searchParams.set(kResultsArg, "1");
+ return copyUrl.toString();
+ },
+ });
+ clipboard.on("success", function (e) {
+ // Focus the input
+
+ // button target
+ const button = e.trigger;
+ const icon = button.querySelector("i.bi");
+
+ // flash "checked"
+ icon.classList.add(checkIcon);
+ icon.classList.remove(linkIcon);
+ setTimeout(function () {
+ icon.classList.remove(checkIcon);
+ icon.classList.add(linkIcon);
+ }, 1000);
+ });
+ }
+
+ // If there is a query, show the link icon
+ if (copyButtonEl) {
+ if (lastQuery && options["copy-button"]) {
+ copyButtonEl.style.display = "flex";
+ } else {
+ copyButtonEl.style.display = "none";
+ }
+ }
+ }
+}
+
+/* Search Index Handling */
+// create the index
+var fuseIndex = undefined;
+var shownWarning = false;
+
+// fuse index options
+const kFuseIndexOptions = {
+ keys: [
+ { name: "title", weight: 20 },
+ { name: "section", weight: 20 },
+ { name: "text", weight: 10 },
+ ],
+ ignoreLocation: true,
+ threshold: 0.1,
+};
+
+async function readSearchData() {
+ // Initialize the search index on demand
+ if (fuseIndex === undefined) {
+ if (window.location.protocol === "file:" && !shownWarning) {
+ window.alert(
+ "Search requires JavaScript features disabled when running in file://... URLs. In order to use search, please run this document in a web server."
+ );
+ shownWarning = true;
+ return;
+ }
+ const fuse = new window.Fuse([], kFuseIndexOptions);
+
+ // fetch the main search.json
+ const response = await fetch(offsetURL("search.json"));
+ if (response.status == 200) {
+ return response.json().then(function (searchDocs) {
+ searchDocs.forEach(function (searchDoc) {
+ fuse.add(searchDoc);
+ });
+ fuseIndex = fuse;
+ return fuseIndex;
+ });
+ } else {
+ return Promise.reject(
+ new Error(
+ "Unexpected status from search index request: " + response.status
+ )
+ );
+ }
+ }
+
+ return fuseIndex;
+}
+
+function inputElement() {
+ return window.document.body.querySelector(".aa-Form .aa-Input");
+}
+
+function focusSearchInput() {
+ setTimeout(() => {
+ const inputEl = inputElement();
+ if (inputEl) {
+ inputEl.focus();
+ }
+ }, 50);
+}
+
+/* Panels */
+const kItemTypeDoc = "document";
+const kItemTypeMore = "document-more";
+const kItemTypeItem = "document-item";
+const kItemTypeError = "error";
+
+function renderItem(
+ item,
+ createElement,
+ state,
+ setActiveItemId,
+ setContext,
+ refresh,
+ quartoSearchOptions
+) {
+ switch (item.type) {
+ case kItemTypeDoc:
+ return createDocumentCard(
+ createElement,
+ "file-richtext",
+ item.title,
+ item.section,
+ item.text,
+ item.href,
+ item.crumbs,
+ quartoSearchOptions
+ );
+ case kItemTypeMore:
+ return createMoreCard(
+ createElement,
+ item,
+ state,
+ setActiveItemId,
+ setContext,
+ refresh
+ );
+ case kItemTypeItem:
+ return createSectionCard(
+ createElement,
+ item.section,
+ item.text,
+ item.href
+ );
+ case kItemTypeError:
+ return createErrorCard(createElement, item.title, item.text);
+ default:
+ return undefined;
+ }
+}
+
+function createDocumentCard(
+ createElement,
+ icon,
+ title,
+ section,
+ text,
+ href,
+ crumbs,
+ quartoSearchOptions
+) {
+ const iconEl = createElement("i", {
+ class: `bi bi-${icon} search-result-icon`,
+ });
+ const titleEl = createElement("p", { class: "search-result-title" }, title);
+ const titleContents = [iconEl, titleEl];
+ const showParent = quartoSearchOptions["show-item-context"];
+ if (crumbs && showParent) {
+ let crumbsOut = undefined;
+ const crumbClz = ["search-result-crumbs"];
+ if (showParent === "root") {
+ crumbsOut = crumbs.length > 1 ? crumbs[0] : undefined;
+ } else if (showParent === "parent") {
+ crumbsOut = crumbs.length > 1 ? crumbs[crumbs.length - 2] : undefined;
+ } else {
+ crumbsOut = crumbs.length > 1 ? crumbs.join(" > ") : undefined;
+ crumbClz.push("search-result-crumbs-wrap");
+ }
+
+ const crumbEl = createElement(
+ "p",
+ { class: crumbClz.join(" ") },
+ crumbsOut
+ );
+ titleContents.push(crumbEl);
+ }
+
+ const titleContainerEl = createElement(
+ "div",
+ { class: "search-result-title-container" },
+ titleContents
+ );
+
+ const textEls = [];
+ if (section) {
+ const sectionEl = createElement(
+ "p",
+ { class: "search-result-section" },
+ section
+ );
+ textEls.push(sectionEl);
+ }
+ const descEl = createElement("p", {
+ class: "search-result-text",
+ dangerouslySetInnerHTML: {
+ __html: text,
+ },
+ });
+ textEls.push(descEl);
+
+ const textContainerEl = createElement(
+ "div",
+ { class: "search-result-text-container" },
+ textEls
+ );
+
+ const containerEl = createElement(
+ "div",
+ {
+ class: "search-result-container",
+ },
+ [titleContainerEl, textContainerEl]
+ );
+
+ const linkEl = createElement(
+ "a",
+ {
+ href: offsetURL(href),
+ class: "search-result-link",
+ },
+ containerEl
+ );
+
+ const classes = ["search-result-doc", "search-item"];
+ if (!section) {
+ classes.push("document-selectable");
+ }
+
+ return createElement(
+ "div",
+ {
+ class: classes.join(" "),
+ },
+ linkEl
+ );
+}
+
+function createMoreCard(
+ createElement,
+ item,
+ state,
+ setActiveItemId,
+ setContext,
+ refresh
+) {
+ const moreCardEl = createElement(
+ "div",
+ {
+ class: "search-result-more search-item",
+ onClick: (e) => {
+ // Handle expanding the sections by adding the expanded
+ // section to the list of expanded sections
+ toggleExpanded(item, state, setContext, setActiveItemId, refresh);
+ e.stopPropagation();
+ },
+ },
+ item.title
+ );
+
+ return moreCardEl;
+}
+
+function toggleExpanded(item, state, setContext, setActiveItemId, refresh) {
+ const expanded = state.context.expanded || [];
+ if (expanded.includes(item.target)) {
+ setContext({
+ expanded: expanded.filter((target) => target !== item.target),
+ });
+ } else {
+ setContext({ expanded: [...expanded, item.target] });
+ }
+
+ refresh();
+ setActiveItemId(item.__autocomplete_id);
+}
+
+function createSectionCard(createElement, section, text, href) {
+ const sectionEl = createSection(createElement, section, text, href);
+ return createElement(
+ "div",
+ {
+ class: "search-result-doc-section search-item",
+ },
+ sectionEl
+ );
+}
+
+function createSection(createElement, title, text, href) {
+ const descEl = createElement("p", {
+ class: "search-result-text",
+ dangerouslySetInnerHTML: {
+ __html: text,
+ },
+ });
+
+ const titleEl = createElement("p", { class: "search-result-section" }, title);
+ const linkEl = createElement(
+ "a",
+ {
+ href: offsetURL(href),
+ class: "search-result-link",
+ },
+ [titleEl, descEl]
+ );
+ return linkEl;
+}
+
+function createErrorCard(createElement, title, text) {
+ const descEl = createElement("p", {
+ class: "search-error-text",
+ dangerouslySetInnerHTML: {
+ __html: text,
+ },
+ });
+
+ const titleEl = createElement("p", {
+ class: "search-error-title",
+ dangerouslySetInnerHTML: {
+ __html: ` ${title}`,
+ },
+ });
+ const errorEl = createElement("div", { class: "search-error" }, [
+ titleEl,
+ descEl,
+ ]);
+ return errorEl;
+}
+
+function positionPanel(pos) {
+ const panelEl = window.document.querySelector(
+ "#quarto-search-results .aa-Panel"
+ );
+ const inputEl = window.document.querySelector(
+ "#quarto-search .aa-Autocomplete"
+ );
+
+ if (panelEl && inputEl) {
+ panelEl.style.top = `${Math.round(panelEl.offsetTop)}px`;
+ if (pos === "start") {
+ panelEl.style.left = `${Math.round(inputEl.left)}px`;
+ } else {
+ panelEl.style.right = `${Math.round(inputEl.offsetRight)}px`;
+ }
+ }
+}
+
+/* Highlighting */
+// highlighting functions
+function highlightMatch(query, text) {
+ if (text) {
+ const start = text.toLowerCase().indexOf(query.toLowerCase());
+ if (start !== -1) {
+ const startMark = "";
+ const endMark = "";
+
+ const end = start + query.length;
+ text =
+ text.slice(0, start) +
+ startMark +
+ text.slice(start, end) +
+ endMark +
+ text.slice(end);
+ const startInfo = clipStart(text, start);
+ const endInfo = clipEnd(
+ text,
+ startInfo.position + startMark.length + endMark.length
+ );
+ text =
+ startInfo.prefix +
+ text.slice(startInfo.position, endInfo.position) +
+ endInfo.suffix;
+
+ return text;
+ } else {
+ return text;
+ }
+ } else {
+ return text;
+ }
+}
+
+function clipStart(text, pos) {
+ const clipStart = pos - 50;
+ if (clipStart < 0) {
+ // This will just return the start of the string
+ return {
+ position: 0,
+ prefix: "",
+ };
+ } else {
+ // We're clipping before the start of the string, walk backwards to the first space.
+ const spacePos = findSpace(text, pos, -1);
+ return {
+ position: spacePos.position,
+ prefix: "",
+ };
+ }
+}
+
+function clipEnd(text, pos) {
+ const clipEnd = pos + 200;
+ if (clipEnd > text.length) {
+ return {
+ position: text.length,
+ suffix: "",
+ };
+ } else {
+ const spacePos = findSpace(text, clipEnd, 1);
+ return {
+ position: spacePos.position,
+ suffix: spacePos.clipped ? "…" : "",
+ };
+ }
+}
+
+function findSpace(text, start, step) {
+ let stepPos = start;
+ while (stepPos > -1 && stepPos < text.length) {
+ const char = text[stepPos];
+ if (char === " " || char === "," || char === ":") {
+ return {
+ position: step === 1 ? stepPos : stepPos - step,
+ clipped: stepPos > 1 && stepPos < text.length,
+ };
+ }
+ stepPos = stepPos + step;
+ }
+
+ return {
+ position: stepPos - step,
+ clipped: false,
+ };
+}
+
+// removes highlighting as implemented by the mark tag
+function clearHighlight(searchterm, el) {
+ const childNodes = el.childNodes;
+ for (let i = childNodes.length - 1; i >= 0; i--) {
+ const node = childNodes[i];
+ if (node.nodeType === Node.ELEMENT_NODE) {
+ if (
+ node.tagName === "MARK" &&
+ node.innerText.toLowerCase() === searchterm.toLowerCase()
+ ) {
+ el.replaceChild(document.createTextNode(node.innerText), node);
+ } else {
+ clearHighlight(searchterm, node);
+ }
+ }
+ }
+}
+
+function escapeRegExp(string) {
+ return string.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"); // $& means the whole matched string
+}
+
+// highlight matches
+function highlight(term, el) {
+ const termRegex = new RegExp(term, "ig");
+ const childNodes = el.childNodes;
+
+ // walk back to front avoid mutating elements in front of us
+ for (let i = childNodes.length - 1; i >= 0; i--) {
+ const node = childNodes[i];
+
+ if (node.nodeType === Node.TEXT_NODE) {
+ // Search text nodes for text to highlight
+ const text = node.nodeValue;
+
+ let startIndex = 0;
+ let matchIndex = text.search(termRegex);
+ if (matchIndex > -1) {
+ const markFragment = document.createDocumentFragment();
+ while (matchIndex > -1) {
+ const prefix = text.slice(startIndex, matchIndex);
+ markFragment.appendChild(document.createTextNode(prefix));
+
+ const mark = document.createElement("mark");
+ mark.appendChild(
+ document.createTextNode(
+ text.slice(matchIndex, matchIndex + term.length)
+ )
+ );
+ markFragment.appendChild(mark);
+
+ startIndex = matchIndex + term.length;
+ matchIndex = text.slice(startIndex).search(new RegExp(term, "ig"));
+ if (matchIndex > -1) {
+ matchIndex = startIndex + matchIndex;
+ }
+ }
+ if (startIndex < text.length) {
+ markFragment.appendChild(
+ document.createTextNode(text.slice(startIndex, text.length))
+ );
+ }
+
+ el.replaceChild(markFragment, node);
+ }
+ } else if (node.nodeType === Node.ELEMENT_NODE) {
+ // recurse through elements
+ highlight(term, node);
+ }
+ }
+}
+
+/* Link Handling */
+// get the offset from this page for a given site root relative url
+function offsetURL(url) {
+ var offset = getMeta("quarto:offset");
+ return offset ? offset + url : url;
+}
+
+// read a meta tag value
+function getMeta(metaName) {
+ var metas = window.document.getElementsByTagName("meta");
+ for (let i = 0; i < metas.length; i++) {
+ if (metas[i].getAttribute("name") === metaName) {
+ return metas[i].getAttribute("content");
+ }
+ }
+ return "";
+}
+
+function algoliaSearch(query, limit, algoliaOptions) {
+ const { getAlgoliaResults } = window["@algolia/autocomplete-preset-algolia"];
+
+ const applicationId = algoliaOptions["application-id"];
+ const searchOnlyApiKey = algoliaOptions["search-only-api-key"];
+ const indexName = algoliaOptions["index-name"];
+ const indexFields = algoliaOptions["index-fields"];
+ const searchClient = window.algoliasearch(applicationId, searchOnlyApiKey);
+ const searchParams = algoliaOptions["params"];
+ const searchAnalytics = !!algoliaOptions["analytics-events"];
+
+ return getAlgoliaResults({
+ searchClient,
+ queries: [
+ {
+ indexName: indexName,
+ query,
+ params: {
+ hitsPerPage: limit,
+ clickAnalytics: searchAnalytics,
+ ...searchParams,
+ },
+ },
+ ],
+ transformResponse: (response) => {
+ if (!indexFields) {
+ return response.hits.map((hit) => {
+ return hit.map((item) => {
+ return {
+ ...item,
+ text: highlightMatch(query, item.text),
+ };
+ });
+ });
+ } else {
+ const remappedHits = response.hits.map((hit) => {
+ return hit.map((item) => {
+ const newItem = { ...item };
+ ["href", "section", "title", "text", "crumbs"].forEach(
+ (keyName) => {
+ const mappedName = indexFields[keyName];
+ if (
+ mappedName &&
+ item[mappedName] !== undefined &&
+ mappedName !== keyName
+ ) {
+ newItem[keyName] = item[mappedName];
+ delete newItem[mappedName];
+ }
+ }
+ );
+ newItem.text = highlightMatch(query, newItem.text);
+ return newItem;
+ });
+ });
+ return remappedHits;
+ }
+ },
+ });
+}
+
+let subSearchTerm = undefined;
+let subSearchFuse = undefined;
+const kFuseMaxWait = 125;
+
+async function fuseSearch(query, fuse, fuseOptions) {
+ let index = fuse;
+ // Fuse.js using the Bitap algorithm for text matching which runs in
+ // O(nm) time (no matter the structure of the text). In our case this
+ // means that long search terms mixed with large index gets very slow
+ //
+ // This injects a subIndex that will be used once the terms get long enough
+ // Usually making this subindex is cheap since there will typically be
+ // a subset of results matching the existing query
+ if (subSearchFuse !== undefined && query.startsWith(subSearchTerm)) {
+ // Use the existing subSearchFuse
+ index = subSearchFuse;
+ } else if (subSearchFuse !== undefined) {
+ // The term changed, discard the existing fuse
+ subSearchFuse = undefined;
+ subSearchTerm = undefined;
+ }
+
+ // Search using the active fuse
+ const then = performance.now();
+ const resultsRaw = await index.search(query, fuseOptions);
+ const now = performance.now();
+
+ const results = resultsRaw.map((result) => {
+ const addParam = (url, name, value) => {
+ const anchorParts = url.split("#");
+ const baseUrl = anchorParts[0];
+ const sep = baseUrl.search("\\?") > 0 ? "&" : "?";
+ anchorParts[0] = baseUrl + sep + name + "=" + value;
+ return anchorParts.join("#");
+ };
+
+ return {
+ title: result.item.title,
+ section: result.item.section,
+ href: addParam(result.item.href, kQueryArg, query),
+ text: highlightMatch(query, result.item.text),
+ crumbs: result.item.crumbs,
+ };
+ });
+
+ // If we don't have a subfuse and the query is long enough, go ahead
+ // and create a subfuse to use for subsequent queries
+ if (
+ now - then > kFuseMaxWait &&
+ subSearchFuse === undefined &&
+ resultsRaw.length < fuseOptions.limit
+ ) {
+ subSearchTerm = query;
+ subSearchFuse = new window.Fuse([], kFuseIndexOptions);
+ resultsRaw.forEach((rr) => {
+ subSearchFuse.add(rr.item);
+ });
+ }
+ return results;
+}
diff --git a/blog/docs/styles.css b/blog/docs/styles.css
new file mode 100644
index 0000000..2ddf50c
--- /dev/null
+++ b/blog/docs/styles.css
@@ -0,0 +1 @@
+/* css styles */
diff --git a/blog/index.qmd b/blog/index.qmd
new file mode 100644
index 0000000..d690afc
--- /dev/null
+++ b/blog/index.qmd
@@ -0,0 +1,18 @@
+---
+# title: "OpenML Blog"
+listing:
+ contents: posts
+ sort: "date desc"
+ type: default
+ categories: true
+ sort-ui: true
+ filter-ui: false
+page-layout: full
+title-block-banner: true
+---
+
+
+# Welcome to the Open Machine Learning blog
+This blog brings you stories about OpenML: why we want to streamline machine learning research, how you can use it, and what we are doing.
+We are also open to [your stories](https://github.com/openml/blog#how-to-contribute) about anything related to open machine learning research, education, and applications.
+Join the conversation :).
diff --git a/blog/openml-logo.png b/blog/openml-logo.png
new file mode 100644
index 0000000..904e91a
Binary files /dev/null and b/blog/openml-logo.png differ
diff --git a/blog/posts/2016-05-02-OpenML/2016-05-02-OpenML.qmd b/blog/posts/2016-05-02-OpenML/2016-05-02-OpenML.qmd
new file mode 100644
index 0000000..b976414
--- /dev/null
+++ b/blog/posts/2016-05-02-OpenML/2016-05-02-OpenML.qmd
@@ -0,0 +1,73 @@
+---
+toc: true
+layout: post
+description: Introduction to OpenML
+date: 2016-05-02
+categories: [OpenML]
+title: OpenML
+image: openml-logo.png
+author: Heidi Seibold
+---
+
+
+[OpenML](http://www.openml.org/) is a very cool new online platform that aims at improving — as the name says — Open Machine Learning. It stands for Open Data, Open Algorithms and Open Research. OpenML is still in it’s beta phase, but already pretty awesome.
+
+With this blog post I would like to introduce the main concepts, show who should be interested in the platform and I will go a little into a challenge it faces.
+
+# Concepts
+
+The following four concepts form the basis of the platform:
+
+- data
+
+- task
+
+- flow
+
+- run
+
+The figure shows how they are connected.
+
+
+
+# Who can make use of OpenML?
+
+#### The domain scientist
+
+You have data that you do not know how to analyse best? Upload your data to OpenML and you will have the whole world helping you. Write a good data and task description to make sure people understand the problem.
+
+#### The data analyst
+
+You like taking part in challenges? Being the best solver of a task? Go to OpenML and check out the many tasks and go solve!
+
+#### The algorithm developer
+
+You developed a statistical method or a machine learning algorithm and want to try it out? You will find plenty of data sets and the possibility to make your algorithm public.
+
+#### The student
+
+You study statistics, data science, machine learning? You want to know what is out there? On OpenML you will find a wide variety of algorithms and, if the solvers do a good job, info on software and implementation.
+
+#### The teacher
+
+You teach a machine learning class and want the students to participate in a challenge? Make up your own task and let the students try solving it. The platform shows who uploaded what and when.
+
+#### The unknown
+
+There are possibly many other people who will benefit from the platform, like meta analysts, benchmarkers and people I can not think of right now.
+
+# How to use OpenML
+
+Other than just browsing the website you can access OpenML through quite some interfaces such as [R](http://www.openml.org/guide#!r) or [WEKA](http://www.openml.org/guide#!plugin_weka). For an example on how to use the R interface check out the [tutorial](http://openml.github.io/openml-r/vignettes/OpenML.html).
+
+The whole project is of course open source. Check out the different [git repositories](https://github.com/openml) for all the code and in case you have any complaints.
+
+# The overfitting problem
+
+Platforms like [kaggle](https://www.kaggle.com/), [crowdanalytics](https://www.crowdanalytix.com/) and [innocentive](https://www.innocentive.com/) host challenges and give people only part of the data so they can evaluate the performance of the algorithm on a separate data set to (try to) prevent overfitting. So far OpenML does not do that. It always shows all the data, and algorithms are evaluated via resampling procedures (on OpenML called estimation procedures). There are big discussions about how to solve the problem of overfitting on OpenML. They go from keeping part of the data hidden for a certain time in the beginning to doing repeated cross-validation on the (overly) good performing flows on a given task. If you have ideas here, please don’t hesitate to leave me a comment.
+
+The platform is still in it’s childhood and may not be perfect yet (If you find issues, post them on the [github page](https://github.com/openml)). But I think it can grow to be a great thing one day.
+
+---
+
+_Originally published at [heidiseibold.github.io](http://heidiseibold.github.io/2016/05/02/OpenML/) on May 2, 2016._
diff --git a/blog/posts/2016-05-02-OpenML/openml-logo.png b/blog/posts/2016-05-02-OpenML/openml-logo.png
new file mode 100644
index 0000000..904e91a
Binary files /dev/null and b/blog/posts/2016-05-02-OpenML/openml-logo.png differ
diff --git a/blog/posts/2016-09-18-mlr-loves-OpenML/2016-09-18-mlr-loves-OpenML.qmd b/blog/posts/2016-09-18-mlr-loves-OpenML/2016-09-18-mlr-loves-OpenML.qmd
new file mode 100644
index 0000000..499d2ab
--- /dev/null
+++ b/blog/posts/2016-09-18-mlr-loves-OpenML/2016-09-18-mlr-loves-OpenML.qmd
@@ -0,0 +1,109 @@
+---
+toc: true
+layout: post
+description: OpenML with mlr
+categories: [OpenML, mlr, R]
+title: mlr loves OpenML
+image: mlropenml.png
+author: Heidi Seibold
+date : 2016-09-18
+---
+
+[OpenML](http://www.openml.org/) stands for Open Machine Learning and is an online platform, which aims at supporting collaborative machine learning online. It is an Open Science project that allows its users to share data, code and machine learning experiments.
+
+At the time of writing this blog post I am in Eindhoven at an [OpenML workshop](http://openml2016dev.openml.org/), where developers and scientists meet to work on improving the project. Some of these people are R users and they (we) are developing an [R package](https://github.com/openml/openml-r) that communicates with the OpenML platform.
+
+
+
+# OpenML in R
+
+The OpenML R package can list and download data sets and machine learning tasks (prediction challenges). In R one can run algorithms on the these data sets/tasks and then upload the results to OpenML. After successful uploading, the website shows how well the algorithm performs. To run the algorithm on a given task the OpenML R package builds on the [mlr](https://github.com/mlr-org/mlr) package. mlr understands what a task is and can run learners on that task. So all the OpenML package needs to do is convert the OpenML objects to objects mlr understands and then mlr deals with the learning.
+
+# A small case study
+
+We want to create a little study on the [OpenML website](http://www.openml.org/), in which we compare different types of Support Vector Machines. The study gets an ID assigned to it, which in our case is 27. We use the function ksvm (with different settings of the function argument type) from package kernlab, which is integrated in mlr (“classif.ksvm”).
+
+
+
+For details on installing and setting up the OpenML R package please see the [guide](https://github.com/openml/openml-r) on GitHub.
+
+Let’s start conducting the study:
+
+- Load the packages and list all tasks which have between 100 and 500 observations.
+
+```
+library("OpenML") library("mlr") library("farff") library("BBmisc")
+
+dsize = c(100, 500) taskinfo_all = listOMLTasks(number.of.instances = dsize)
+```
+
+- Select all supervised classification tasks that do 10-fold cross-validation and choose only one task per data set. To keep the study simple and fast to compute, select only the first three tasks.
+
+```
+taskinfo_10cv = subset(taskinfo_all,
+ task.type == "Supervised Classification" &
+ estimation.procedure == "10-fold Crossvalidation" &
+ evaluation.measures == "predictive_accuracy" &
+ number.of.missing.values == 0 &
+ number.of.classes %in% c(2, 4))
+
+taskinfo = taskinfo_10cv[1:3, ]
+```
+
+- Create the learners we want to compare.
+
+```
+lrn.list = list(
+ makeLearner("classif.ksvm", type = "C-svc"),
+ makeLearner("classif.ksvm", type = "kbb-svc"),
+ makeLearner("classif.ksvm", type = "spoc-svc")
+)
+```
+
+- Run the learners on the three tasks.
+
+```
+grid = expand.grid(task.id = taskinfo$task.id,
+ lrn.ind = seq_along(lrn.list))
+
+runs = lapply(seq_row(grid), function(i) {
+ message(i)
+ task = getOMLTask(grid$task.id[i])
+ ind = grid$lrn.ind[i]
+ runTaskMlr(task, lrn.list[[ind]])
+})
+```
+
+- And finally upload the runs to OpenML. The upload function (uploadOMLRun) returns the ID of the uploaded run object. When uploading runs that are part of a certain study, tag it with study\_ and the study ID. After uploading the runs appear on the website and can be found using the tag or via the [study homepage](http://www.openml.org/index.php/s/27).
+
+```
+## please do not spam the OpenML server by uploading these
+## tasks. I already did that.
+run.id = lapply(runs, uploadOMLRun, tags = "study_27")
+```
+
+- To show the results of our study, list the run evaluations and make a nice plot.
+
+```
+evals = listOMLRunEvaluations(tag = "study_27")
+
+evals$task.id = as.factor(evals$task.id)
+evals$setup.id = as.factor(evals$setup.id)
+
+library("ggplot2")
+ggplot(evals, aes(x = setup.id, y = predictive.accuracy,
+ color = data.name, group = task.id)) +
+ geom_point() + geom_line()
+```
+
+
+
+Now you can go ahead and create a bigger study using the techniques you have learned.
+
+# Further infos
+
+If you are interested in more, check out the OpenML [blog](https://medium.com/open-machine-learning), the [paper](https://www.researchgate.net/publication/263890323_OpenML_Networked_science_in_machine_learning) and the [GitHub repos](https://github.com/openml).
+
+---
+
+_Originally published at [mlr-org.github.io](http://mlr-org.github.io/mlr-loves-OpenML/)._
diff --git a/blog/posts/2016-09-18-mlr-loves-OpenML/mlropenml.png b/blog/posts/2016-09-18-mlr-loves-OpenML/mlropenml.png
new file mode 100644
index 0000000..71a8ee7
Binary files /dev/null and b/blog/posts/2016-09-18-mlr-loves-OpenML/mlropenml.png differ
diff --git a/blog/posts/2017-03-03-Basic-components-of-OpenML/2017-03-03-Basic-components-of-OpenML.qmd b/blog/posts/2017-03-03-Basic-components-of-OpenML/2017-03-03-Basic-components-of-OpenML.qmd
new file mode 100644
index 0000000..a746b43
--- /dev/null
+++ b/blog/posts/2017-03-03-Basic-components-of-OpenML/2017-03-03-Basic-components-of-OpenML.qmd
@@ -0,0 +1,65 @@
+---
+toc: true
+layout: post
+description: Understanding the building blocks of OpenML
+categories: [OpenML]
+title: Basic components of OpenML
+author: Jan van Rijn
+date : 2017-03-03
+---
+
+During my PhD, we developed OpenML, an online experiment database for Machine Learning. Researchers are encouraged to upload their experimental results on it, so that these can be reused by anyone. Various high level papers have been published that overview the design goals, benefits and opportunities (for example, at [ECML/PKDD 2013](http://link.springer.com/chapter/10.1007%2F978-3-642-40994-3_46), [SIGKDD Explorations](http://dl.acm.org/citation.cfm?id=2641198) and [JLMR](http://www.jmlr.org/proceedings/papers/v41/vanschoren15.html)). However, there is no clear overview of the basic components upon which the platform is build. In this blog post I will review these, and discuss some best practises.
+
+# Data
+
+One of the core components of OpenML are datasets. People can upload their datasets, and the system automatically organises these on line. An example of a dataset is the well-known [Iris dataset](http://www.openml.org/d/61). It shows all features, once of these is identified as the ‘default target attribute’, although this concept is flexible. It also shows some automatically computed data qualities (or, meta-features). Each dataset has it’s own unique ID. Information about the dataset, the data features and the data qualities can be obtained automatically by means of the following API functions:
+
+- [Get all available datasets](http://www.openml.org/api_docs/#!/data/get_data_list)
+
+- [Get dataset](http://www.openml.org/api_docs/#!/data/get_data_id) (required the data id)
+
+- [Get data features](http://www.openml.org/api_docs/#!/data/get_data_features_id) (requires the data id)
+
+- [Get data qualities](http://www.openml.org/api_docs/#!/data/get_data_qualities_id) (requires the data id)
+
+# Task types and tasks
+
+A dataset alone does not constitute a scientific task. We must first agree on what types of results are expected to be shared. This is expressed in task types: they define what types of inputs are given, which types of output are expected to be returned, and what protocols should be used. For instance, classification tasks should include well-defined cross-validation procedures, labelled input data, and require predictions as outputs. The collection of all this information together is called a task. The Iris dataset has various tasks defined on it, [for example this one](http://www.openml.org/t/59). Although the web-interface does not show it, this task formally describes the target attribute that should be modelled (in this case the same as the default target attribute of the dataset, but this is flexible), the quality estimation procedure (10-fold cross-validation), the evaluation measure (predictive accuracy) and the cross-validation folds. Useful API operations include:
+
+- [Get all available tasks](http://www.openml.org/api_docs/#!/task/get_task_list)
+
+- [Get all available tasks of a given type](http://www.openml.org/api_docs/#!/task/get_task_list_type_id) (e.g. get all Classification tasks, requires the id of the task type)
+
+- [Get the details of a task](http://www.openml.org/api_docs/#!/task/get_task_id) (requires task id)
+
+Currently, there are a wide range of task types defined on OpenML, including classification, regression, on line learning, clustering and subgroup discovery. Although this set can be extended, this is currently not a supported API operation (meaning that we will add them by hand). If you interested in task types that are currently not supported, please contact us.
+
+# Flows
+
+Tasks can be ‘solved’ by classifiers (or algorithms, workflows, flows). OpenML stores references to these flows. It is important to stress that flows are actually ran on the computer of the user, only meta-information about the flow is stored on OpenML. This information includes basic trivialities such as the creator, toolbox and compilation instructions, but also more formal description about hyper parameter. A flow can also contain subflows, for example, the flow Bagging can have a subflow ‘Decision Tree’ which would make the flow ‘Bagging of Decision Trees’. A flow is distinguished by it’s name and ‘external version’, which are both provided by the uploader. When uploading a flow, it is important to think about a good naming convention for the both, for example, the git commit number could be used as external version, as this uniquely identifies a state of the code. Ideally, when two persons are using the same flow, they will use the same name and external version, so that results of the flows can be compared across tasks. (This is ensured when using the toolboxed in which OpenML is integrated, such as Weka, Scikit Learn and MLR). Useful API functions are:
+
+- [List all flows](http://www.openml.org/api_docs/#!/flow/get_flow_list)
+
+- [List all my flows](http://www.openml.org/api_docs/#!/flow/get_flow_owned)
+
+- [Give details about a given flow ](http://www.openml.org/api_docs/#!/flow/get_flow_id)(requires flow id)
+
+# Runs
+
+Whenever a flow executes a task, this is called a run. The existence of runs is actually the main contribution of OpenML. Some experiments take weeks to complete, and having the results stored on OpenML helps other researchers resuse the experiments. The task description specifies which information should be uploaded in order to have a valid run, in most cases, for each cross-validation fold the predictions on the test set. This allows OpenML to calculate basic evaluation measures, such as predictive accuracy, ROC curves and many more. Also information about the flow and hyper parameter settings should be provided. Some useful API functions:
+
+- [List all runs performed on a given task](http://www.openml.org/api_docs/#!/run/get_run_list_task_ids) (requires task id, e.g., the iris task is 59)
+
+- [Compare two flows on all tasks](http://www.openml.org/api_docs/#!/run/get_run_list_filters) (requires a comma separated list of flow ids, e.g., 1720, 1721 for comparing k-nn with a decision tree)
+
+- And many more …
+
+Usually, the result is in some XML or JSON format (depending on the preference of the user), linking together various task ids, flow ids, etc. In order for this to become meaningful, the user needs to perform other API tasks to get information about what flows were executed, what tasks and datasets were used, etc. Details about this will be provided in another post.
+
+# Setups
+
+Every run that is executed by a flow, contains information about the hyper parameter settings of the flow. A setup is the combination of all parameter settings of a given flow. OpenML internally links the result of a given run to a setup id. This way, experiments can be done across hyper parameter settings. For example,
+
+- [Compare two setups on all tasks](http://www.openml.org/api_docs/#!/run/get_run_list_filters) (requires a comma separated list of setup ids, e.g., 8994, 8995, 8996 for comparing multiple MLP configurations)
+
+As setups constitute a complex concept, most of the operations concerning setups are hidden from the user. Hence, not all setup functions are properly documented yet. A later blogpost will detail on these.
diff --git a/blog/posts/2019-10-24-OpenML-workshop-at-Dagstuhl/2019-10-24-OpenML-workshop-at-Dagstuhl.qmd b/blog/posts/2019-10-24-OpenML-workshop-at-Dagstuhl/2019-10-24-OpenML-workshop-at-Dagstuhl.qmd
new file mode 100644
index 0000000..d0cae5e
--- /dev/null
+++ b/blog/posts/2019-10-24-OpenML-workshop-at-Dagstuhl/2019-10-24-OpenML-workshop-at-Dagstuhl.qmd
@@ -0,0 +1,121 @@
+---
+toc: true
+layout: post
+description: OpenML workshop at Dagstuhl
+categories: [OpenML]
+title: OpenML workshop at Dagstuhl
+image: dagstuhl.jpeg
+author: Heidi Seibold
+date : 2019-10-24
+---
+
+**October 2019**
+
+Twice a year the [OpenML](https://www.openml.org/) community meets for a [workshop/hackathon/unconference](http://meet.openml.org/). We improve the platform, discuss and learn for 5 days. If that sounds interesting to you, [get in touch](https://www.openml.org/contact) or follow updates on [meet.openml.org](http://meet.openml.org/).
+
+This time the workshop took place at [Dagstuhl](https://www.dagstuhl.de/19423), a great place for Seminars.
+
+
+
+
+
+We had several breakouts where workshop attendees can join to learn, discuss and progress OpenML. In the following we discuss some of the topics we touched.
+
+**Science projects**
+
+Brainstorm on scientific projects to do with OpenML. Prioritize impactful, well-defined research ideas.
+We came up with quite a long list of very promising research questions that should actually be quite easy to answer based on OpenML. Many of these are along the lines of empirically providing evidence to verify or bust commonly-held beliefs in the community. Frank would gladly hire a strong postdoc (or quite independent PhD student) and maybe a research engineer, to work on these scientific questions under the umbrella of “evidence-based machine learning with OpenML”.
+
+**Benchmarking using OpenML**
+
+Define guidelines on how to define world-class benchmarks and how to run them.
+
+**Diverse datasets**
+
+New and more diverse datasets.
+
+**Dataset quality**
+
+How to measure data quality and how to improve the quality of datasets on OpenML.
+
+**OpenML use cases for novices**
+
+Shortlist common use cases and start writing accessible blog posts for novice users. If you are a OpenML newbie we need your help with this topic.
+
+The output of this breakout will be at least one blog post. Keep an eye out for them here :)
+
+**Running a competition using OpenML**
+
+We assessed the current issues with running in-class competitions for teaching purposes using OpenML (biggest one: easy use for non-developers) and brainstormed on a new competition format with related competitions, one for each component of a solution, such as HPO, creating good meta-features, creating a good search space, etc.
+
+**Planning future workshops**
+
+Decide on location and timing for the next couple of workshops. The next OpenML workshop will be in Spring (week of March 30th or week of April 14th) close to Munich. For updates check [http://meet.openml.org](http://meet.openml.org/). The workshop will be cohosted with some other Open Source Machine Learning projects.
+
+Furthermore a workshop in Austin is being planned for next summer and a datathon is in planning. We are planning to organize dev sprints at various PyCons next year.
+
+**New frontend**
+
+Feedback session on new frontend, Additional visualization for datasets
+
+**Future of client APIs**
+
+Currently, a lot of resources are bound developing different client APIs, such as the [Python API](https://openml.github.io/openml-python/master/), [R API](https://github.com/openml/openml-r) and [Java API](https://github.com/openml/openml-java). We discussed how we can better share work and code between the different APIs and the server. For now we are working on automatically generating the [Swagger API](https://www.openml.org/api_docs#!/data/get_data_id) documentation from the PHP function documentation, which in turn will allow us to generate (documented) parts of the APIs, reduce the need for maintenance and will help to spread updates on the API faster.
+
+**Flow 2.0 design**
+
+Current flow design used in OpenML was inspired by Weka, but through time many limitations have been identified, primarily that existing flow does not allow duplicate use of same component and that it cannot express DAG-based ML programs. We started working on a new [specification](https://github.com/openml/flow2) building on insights from mlr3 and d3m projects, centered around DAG representation. Current plan is to prepare a draft specification and implement prototype converters between other systems and this new specification. Once we do that we will re-evaluate the amount of work it took to build those prototypes and how well the specification satisfied those other systems.
+
+**Random Bot**
+
+The LRZ in Munich provided us with CPU time during the SuperMUC-NG supercomputer test phase, which we used to perform experiments of popular machine learning algorithms with random hyperparameter configurations. This resulted in millions of data points on more than a hundred datasets that we will analyse and publish. The data can be used to learn about typical behaviour of different learners across different datasets, and to construct surrogate models for tuning algorithm benchmarks.
+
+**AutoML Benchmark (Janek)**
+
+A study was created containing 76 binary and multiclass tasks of reasonable difficulty.
+
+These can be used as a more difficult version of OpenML-100 or in [amlb](https://github.com/openml/automlbenchmark) a platform for reproducible benchmarking of AutoML systems.
+
+**R API**
+
+Short session on how the R api will (need to) change. The main issue discussed was that the OpenML R package runs with mlr and breaks when the new package (mlr3) is loaded. We will update the current OpenML R package to work with mlr3. At the same time we are thinking about a vision for a rewrite of the OpenML R package.
+
+**Python API**
+
+We made a lot of improvements to the Python API over the week, with over 20 PRs merged! We’ve added more examples on how to use the package, fixed bugs, improved documentation and refactored code. In the coming days we’re going to make all these improvements available in a new PyPI release. For those looking for a higher level overview of the package, we will publish a paper next week which highlights use-cases, its software design, and project structure.
+
+**Benchmarking paper**
+
+We are working on a comprehensive paper using sklearn, mlr and WEKA, which should demonstrate how OpenML can be used for proper benchmarking and analysis.
+
+**Guidelines / Overfitting / Comparable Metalearning**
+
+There are plans for writing a guidelines and pitfalls paper on benchmarking, meta-overfitting and statistical analysis of results on OpenML.
+
+**Data Formats**
+
+Currently OpenML supports only tabular data in ARFF data format. This is very limiting for many ML tasks. We discussed and explored other data formats we could use as the future next data format. We will post a separate blog post about our process and insights.
+
+**Funding**
+
+OpenML is looking for funding (developers). New ideas on obtaining funding are very welcome. We discussed some ideas: American funding (we need a collaboration partner); ALICE / CLAIRE; Companies. We are a foundation now, which might make it easier.
+
+Some of the PIs (in particular Bernd Bischl, Frank Hutter and Dawn Song) in the project offer positions with a mix of ML research and development. Contact them if you are interested!
+
+
+
+
+
+We had some talks at the workshop as well:
+
+- **Mitar Milutinovic**: [A short introduction to Data Driven Discovery (D3M)](https://mitar.gitlab.io/d3m-short-introduction-slides/#/)
+
+- **Yiwen Zhu and Markus Weimer**: Large-scale analysis of Jupyter notebooks
+
+- **Martin Binder, Michel Lang, Florian Pfisterer, Bernd Bischl**: [Pipelining with mlr3](https://github.com/mlr-org/mlr-outreach/blob/master/2019_dagstuhl/mlr3pipelines.pdf)
+
+… and lots of fun…
+
+
+
+Wanna join the OpenML community? [Get in touch](https://www.openml.org/contact)!
diff --git a/blog/posts/2019-10-24-OpenML-workshop-at-Dagstuhl/dagstuhl.jpeg b/blog/posts/2019-10-24-OpenML-workshop-at-Dagstuhl/dagstuhl.jpeg
new file mode 100644
index 0000000..daed7e1
Binary files /dev/null and b/blog/posts/2019-10-24-OpenML-workshop-at-Dagstuhl/dagstuhl.jpeg differ
diff --git a/blog/posts/2019-10-26-OpenML-Machine-Learning-as-a-community/2019-10-26-OpenML-Machine-Learning-as-a-community.ipynb b/blog/posts/2019-10-26-OpenML-Machine-Learning-as-a-community/2019-10-26-OpenML-Machine-Learning-as-a-community.ipynb
new file mode 100644
index 0000000..2991cf0
--- /dev/null
+++ b/blog/posts/2019-10-26-OpenML-Machine-Learning-as-a-community/2019-10-26-OpenML-Machine-Learning-as-a-community.ipynb
@@ -0,0 +1,2706 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "---\n",
+ "toc: true\n",
+ "title: OpenML - Machine Learning as a community\n",
+ "description: A description of how OpenML fits into traditional ML practices\n",
+ "badges: true\n",
+ "categories: [OpenML]\n",
+ "image: openml.png\n",
+ "author: Neeratyoy Mallik\n",
+ "date: 2019-10-26\n",
+ "---\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "[OpenML](https://www.openml.org/) is an online Machine Learning (ML) experiments database accessible to everyone for free. The core idea is to have a single repository of datasets and results of ML experiments on them. Despite having gained a lot of popularity in recent years, with a plethora of tools now available, the numerous ML experimentations continue to happen in silos and not necessarily as one whole shared community.\n",
+ "In this post, we shall try to get a brief glimpse of what OpenML offers and how it can fit our current Machine Learning practices.\n",
+ "\n",
+ "Let us jump straight at getting our hands dirty by building a simple machine learning model. If it is simplicity we are looking for, it has to be the Iris dataset that we shall work with. In the example script below, we are going to load the Iris dataset available with scikit-learn, use 10-fold cross-validation to evaluate a Random Forest of 10 trees. Sounds trivial enough and is indeed less than 10 lines of code."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from sklearn import datasets\n",
+ "from sklearn.svm import SVC\n",
+ "from sklearn.ensemble import RandomForestClassifier\n",
+ "from sklearn.model_selection import cross_val_score"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "(150, 4) (150,)\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Loading Iris dataset\n",
+ "X, y = datasets.load_iris(return_X_y=True)\n",
+ "print(X.shape, y.shape)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Initializing a Random Forest with \n",
+ "# arbitrary hyperparameters\n",
+ "# max_depth kept as 2 since Iris has\n",
+ "# only 4 features\n",
+ "clf = RandomForestClassifier(n_estimators=10, max_depth=2)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Mean score : 0.94000\n"
+ ]
+ }
+ ],
+ "source": [
+ "scores = cross_val_score(clf, X, y, cv=5, scoring='accuracy')\n",
+ "print(\"Mean score : {:.5f}\".format(scores.mean()))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "A simple script and we achieve a mean accuracy of **95.33%**. That was easy. It is really amazing how far we have come with ML tools that make it easy to get started. As a result, we have hundreds of thousands of people working with these tools every day. That inevitably leads to the reinvention of the wheel. The tasks that each individual ML practitioner performs often have significant overlaps and can be omitted by reusing what someone from the community has done already. At the end of the day, we didn't build a Random Forest model all the way from scratch. We gladly reused code written by generous folks from the community. The special attribute of our species is the ability to work as a collective wherein our combined intellect becomes larger than the individual sum of parts. Why not do the same for ML? I mean, can I see what other ML practitioners have done to get better scores on the Iris dataset?\n",
+ "\n",
+ "Answering this is one of the targets of this post. We shall subsequently explore if this can be done, with the help of [OpenML](https://www.openml.org/). However, first, we shall briefly familiarize ourselves with few terminologies and see how we can split the earlier example we saw into modular components."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### OpenML Components"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Image source: https://medium.com/open-machine-learning/openml-1e0d43f0ae13"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "**Dataset**: OpenML houses over 2k+ active datasets for various regression, classification, clustering, survival analysis, stream processing tasks and more. Any user can upload a dataset. Once uploaded, the server computes certain meta-features on the dataset - *Number of classes*, *Number of missing values*, *Number of features*, etc. With respect to our earlier example, the following line is the equivalent of fetching a dataset from OpenML."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "X, y = datasets.load_iris(return_X_y=True)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "**Task**: A task is linked to a specific dataset, defining what the target/dependent variable is. Also specifies evaluation measures such as - accuracy, precision, area under curve, etc. or the kind of estimation procedure to be used such as - 10-fold *cross-validation*, n% holdout set, etc. With respect to our earlier example, the *parameters* to the following function call capture the idea of a task."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "scores = cross_val_score(clf, X, y, cv=5, scoring='accuracy')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "**Flow**: Describes the kind of modelling to be performed. It could be a flow or a series of steps, i.e., a scikit-learn pipeline. For now, we have used a simple Random Forest model which is the *flow* component here."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "clf = RandomForestClassifier(n_estimators=10, max_depth=2)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "**Run**: Pairs a *flow* and task together which results in a *run*. The *run* has the predictions which are turned into *evaluations* by the server. This is effectively captured by the *execution* of the line:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "scores = cross_val_score(clf, X, y, cv=5, scoring='accuracy')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now, this may appear a little obfuscating given that we are trying to compartmentalize a simple 10-line code which works just fine. However, if we take a few seconds to go through the 4 components explained above, we can see that it makes our *training of a Random Forest* on Iris a series of modular tasks. Modules are such a fundamental concept in Computer Science. They are like Lego blocks. Once we have modules, it means we can plug and play at ease. The code snippet below attempts to rewrite the earlier example using the ideas of the OpenML components described, to give a glimpse of what we can potentially gain during experimentations."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from sklearn import datasets\n",
+ "from sklearn.svm import SVC\n",
+ "from sklearn.ensemble import RandomForestClassifier\n",
+ "from sklearn.model_selection import cross_val_score"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "#### DATASET component"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# To load IRIS dataset as a dataset module/component\n",
+ "def dataset():\n",
+ " X, y = datasets.load_iris(return_X_y=True)\n",
+ " return X, y"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "#### TASK component"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 11,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Tasks here define the number of cross-validation folds\n",
+ "# and the scoring metric to be used for evaluation\n",
+ "def task_1(f):\n",
+ " X, y = dataset() # loads IRIS\n",
+ " return cross_val_score(f, X, y, cv=5, \n",
+ " scoring='accuracy')\n",
+ "\n",
+ "def task_2(f):\n",
+ " X, y = dataset() # loads IRIS\n",
+ " return cross_val_score(f, X, y, cv=15, \n",
+ " scoring='balanced_accuracy')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "#### FLOW component"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 12,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Flows determine the modelling technique to be applied\n",
+ "# Helps define a model irrespective of dataset or tasks\n",
+ "def flow_1():\n",
+ " clf = RandomForestClassifier(n_estimators=10, max_depth=2)\n",
+ " return clf\n",
+ "\n",
+ "def flow_2():\n",
+ " clf = SVC(gamma='auto', kernel='linear')\n",
+ " return clf "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "#### RUN component"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 13,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Runs essentially evaluates a task-flow pairing \n",
+ "# and therefore in effect executs the modelling \n",
+ "# of a dataset as per the task task definition\n",
+ "def run(task, flow):\n",
+ " return task(flow)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 14,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "RF using task 1: 0.95333; task 2: 0.94444\n",
+ "SVM using task 1: 0.98; task 2: 0.97222\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Results for Random Forest\n",
+ "rf_task_1 = run(task_1, flow_1())\n",
+ "rf_task_2 = run(task_2, flow_1())\n",
+ "print(\"RF using task 1: {:<.5}; task 2: {:<.5}\".format(rf_task_1.mean(), rf_task_2.mean()))\n",
+ "\n",
+ "# Results for SVM\n",
+ "svm_task_1 = run(task_1, flow_2())\n",
+ "svm_task_2 = run(task_2, flow_2())\n",
+ "print(\"SVM using task 1: {:<.5}; task 2: {:<.5}\".format(svm_task_1.mean(), svm_task_2.mean()))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We can, therefore, compose various different tasks, flows, which are independent operations. Runs can then pair any such task and flow to construct an ML *workflow* and return the evaluated scores. This approach can help us define such components one-time, and we can extend this for any combination of a dataset, model, and for any number of evaluations in the future. Imagine if the entire ML *community* defines such tasks and various simple to complicated flows that they use in their daily practice. We can build custom working ML pipeline and even get to compare performances of our techniques on the same *task* with others! OpenML aims exactly for that. In the next section of this post, we shall scratch the surface of OpenML to see if we can actually do with OpenML what it promises."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Using OpenML"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "OpenML-Python can be installed using *pip* or by [cloning the git repo](https://openml.github.io/openml-python/develop/contributing.html#installation) and installing the current development version. So shall we then install OpenML? ;) It will be beneficial if the code snippets are tried out as this post is read. A consolidated Jupyter notebook with all the code can be found [here](https://nbviewer.jupyter.org/github/Neeratyoy/openml-python/blob/blog/OpenML%20-%20Machine%20Learning%20as%20a%20community.ipynb).\n",
+ "\n",
+ "Now that we have OpenML, let us jump straight into figuring out how we can get the Iris dataset from there. We can always browse the[OpenML website](https://www.openml.org/) and search for Iris. That is the easy route. Let us get familiar with the programmatic approach and learn how to fish instead. The OpenML-Python API can be found [here](https://openml.github.io/openml-python/develop/api.html)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "#### Retrieving Iris from OpenML"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "In the example below, we will list out all possible datasets available in OpenML. We can choose the output format. I'll go with *dataframe* so that we obtain a pandas DataFrame and can get a neat tabular representation to search and sort specific entries."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 15,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import openml\n",
+ "\n",
+ "import numpy as np\n",
+ "import pandas as pd"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 16,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "(3073, 16)\n",
+ "did\n",
+ "name\n",
+ "version\n",
+ "uploader\n",
+ "status\n",
+ "format\n",
+ "MajorityClassSize\n",
+ "MaxNominalAttDistinctValues\n",
+ "MinorityClassSize\n",
+ "NumberOfClasses\n",
+ "NumberOfFeatures\n",
+ "NumberOfInstances\n",
+ "NumberOfInstancesWithMissingValues\n",
+ "NumberOfMissingValues\n",
+ "NumberOfNumericFeatures\n",
+ "NumberOfSymbolicFeatures\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Fetching the list of all available datasets on OpenML\n",
+ "d = openml.datasets.list_datasets(output_format='dataframe')\n",
+ "print(d.shape)\n",
+ "\n",
+ "# Listing column names or attributes that OpenML offers\n",
+ "for name in d.columns:\n",
+ " print(name)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 17,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ " did name version uploader status format MajorityClassSize \\\n",
+ "2 2 anneal 1 1 active ARFF 684.0 \n",
+ "3 3 kr-vs-kp 1 1 active ARFF 1669.0 \n",
+ "4 4 labor 1 1 active ARFF 37.0 \n",
+ "5 5 arrhythmia 1 1 active ARFF 245.0 \n",
+ "6 6 letter 1 1 active ARFF 813.0 \n",
+ "\n",
+ " MaxNominalAttDistinctValues MinorityClassSize NumberOfClasses \\\n",
+ "2 7.0 8.0 5.0 \n",
+ "3 3.0 1527.0 2.0 \n",
+ "4 3.0 20.0 2.0 \n",
+ "5 13.0 2.0 13.0 \n",
+ "6 26.0 734.0 26.0 \n",
+ "\n",
+ " NumberOfFeatures NumberOfInstances NumberOfInstancesWithMissingValues \\\n",
+ "2 39.0 898.0 898.0 \n",
+ "3 37.0 3196.0 0.0 \n",
+ "4 17.0 57.0 56.0 \n",
+ "5 280.0 452.0 384.0 \n",
+ "6 17.0 20000.0 0.0 \n",
+ "\n",
+ " NumberOfMissingValues NumberOfNumericFeatures NumberOfSymbolicFeatures \n",
+ "2 22175.0 6.0 33.0 \n",
+ "3 0.0 0.0 37.0 \n",
+ "4 326.0 8.0 9.0 \n",
+ "5 408.0 206.0 74.0 \n",
+ "6 0.0 16.0 1.0 \n"
+ ]
+ }
+ ],
+ "source": [
+ "print(d.head())"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The column names indicate that they contain the meta-information about each of the datasets, and at this instance, we have access to **2958** datasets as indicated by the shape of the dataframe. We shall try searching for 'iris' in the column *name* and also use the *version* column to sort the results."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 18,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "| \n", + " | did | \n", + "name | \n", + "version | \n", + "uploader | \n", + "status | \n", + "format | \n", + "MajorityClassSize | \n", + "MaxNominalAttDistinctValues | \n", + "MinorityClassSize | \n", + "NumberOfClasses | \n", + "NumberOfFeatures | \n", + "NumberOfInstances | \n", + "NumberOfInstancesWithMissingValues | \n", + "NumberOfMissingValues | \n", + "NumberOfNumericFeatures | \n", + "NumberOfSymbolicFeatures | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 61 | \n", + "61 | \n", + "iris | \n", + "1 | \n", + "1 | \n", + "active | \n", + "ARFF | \n", + "50.0 | \n", + "3.0 | \n", + "50.0 | \n", + "3.0 | \n", + "5.0 | \n", + "150.0 | \n", + "0.0 | \n", + "0.0 | \n", + "4.0 | \n", + "1.0 | \n", + "
| 41950 | \n", + "41950 | \n", + "iris_test_upload | \n", + "1 | \n", + "4030 | \n", + "active | \n", + "ARFF | \n", + "50.0 | \n", + "3.0 | \n", + "50.0 | \n", + "3.0 | \n", + "5.0 | \n", + "150.0 | \n", + "0.0 | \n", + "0.0 | \n", + "4.0 | \n", + "1.0 | \n", + "
| 42261 | \n", + "42261 | \n", + "iris-example | \n", + "1 | \n", + "348 | \n", + "active | \n", + "ARFF | \n", + "50.0 | \n", + "NaN | \n", + "50.0 | \n", + "3.0 | \n", + "5.0 | \n", + "150.0 | \n", + "0.0 | \n", + "0.0 | \n", + "4.0 | \n", + "1.0 | \n", + "
| 451 | \n", + "451 | \n", + "irish | \n", + "1 | \n", + "2 | \n", + "active | \n", + "ARFF | \n", + "278.0 | \n", + "10.0 | \n", + "222.0 | \n", + "2.0 | \n", + "6.0 | \n", + "500.0 | \n", + "32.0 | \n", + "32.0 | \n", + "2.0 | \n", + "4.0 | \n", + "
| 969 | \n", + "969 | \n", + "iris | \n", + "3 | \n", + "2 | \n", + "active | \n", + "ARFF | \n", + "100.0 | \n", + "2.0 | \n", + "50.0 | \n", + "2.0 | \n", + "5.0 | \n", + "150.0 | \n", + "0.0 | \n", + "0.0 | \n", + "4.0 | \n", + "1.0 | \n", + "
| \n", + " | tid | \n", + "ttid | \n", + "did | \n", + "name | \n", + "task_type | \n", + "status | \n", + "estimation_procedure | \n", + "evaluation_measures | \n", + "source_data | \n", + "target_feature | \n", + "... | \n", + "NumberOfFeatures | \n", + "NumberOfInstances | \n", + "NumberOfInstancesWithMissingValues | \n", + "NumberOfMissingValues | \n", + "NumberOfNumericFeatures | \n", + "NumberOfSymbolicFeatures | \n", + "number_samples | \n", + "cost_matrix | \n", + "quality_measure | \n", + "target_value | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 59 | \n", + "59 | \n", + "1 | \n", + "61 | \n", + "iris | \n", + "Supervised Classification | \n", + "active | \n", + "10-fold Crossvalidation | \n", + "predictive_accuracy | \n", + "61 | \n", + "class | \n", + "... | \n", + "5 | \n", + "150 | \n", + "0 | \n", + "0 | \n", + "4 | \n", + "1 | \n", + "NaN | \n", + "NaN | \n", + "NaN | \n", + "NaN | \n", + "
| 118 | \n", + "118 | \n", + "3 | \n", + "61 | \n", + "iris | \n", + "Learning Curve | \n", + "active | \n", + "10 times 10-fold Learning Curve | \n", + "predictive_accuracy | \n", + "61 | \n", + "class | \n", + "... | \n", + "5 | \n", + "150 | \n", + "0 | \n", + "0 | \n", + "4 | \n", + "1 | \n", + "4 | \n", + "NaN | \n", + "NaN | \n", + "NaN | \n", + "
| 289 | \n", + "289 | \n", + "1 | \n", + "61 | \n", + "iris | \n", + "Supervised Classification | \n", + "active | \n", + "33% Holdout set | \n", + "predictive_accuracy | \n", + "61 | \n", + "class | \n", + "... | \n", + "5 | \n", + "150 | \n", + "0 | \n", + "0 | \n", + "4 | \n", + "1 | \n", + "NaN | \n", + "NaN | \n", + "NaN | \n", + "NaN | \n", + "
| 1758 | \n", + "1758 | \n", + "3 | \n", + "61 | \n", + "iris | \n", + "Learning Curve | \n", + "active | \n", + "10-fold Learning Curve | \n", + "predictive_accuracy | \n", + "61 | \n", + "class | \n", + "... | \n", + "5 | \n", + "150 | \n", + "0 | \n", + "0 | \n", + "4 | \n", + "1 | \n", + "4 | \n", + "NaN | \n", + "NaN | \n", + "NaN | \n", + "
| 1823 | \n", + "1823 | \n", + "1 | \n", + "61 | \n", + "iris | \n", + "Supervised Classification | \n", + "active | \n", + "5 times 2-fold Crossvalidation | \n", + "predictive_accuracy | \n", + "61 | \n", + "class | \n", + "... | \n", + "5 | \n", + "150 | \n", + "0 | \n", + "0 | \n", + "4 | \n", + "1 | \n", + "NaN | \n", + "NaN | \n", + "NaN | \n", + "NaN | \n", + "
5 rows × 24 columns
\n", + "| \n", + " | tid | \n", + "ttid | \n", + "did | \n", + "name | \n", + "task_type | \n", + "status | \n", + "estimation_procedure | \n", + "evaluation_measures | \n", + "source_data | \n", + "target_feature | \n", + "... | \n", + "NumberOfFeatures | \n", + "NumberOfInstances | \n", + "NumberOfInstancesWithMissingValues | \n", + "NumberOfMissingValues | \n", + "NumberOfNumericFeatures | \n", + "NumberOfSymbolicFeatures | \n", + "number_samples | \n", + "cost_matrix | \n", + "quality_measure | \n", + "target_value | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 59 | \n", + "59 | \n", + "1 | \n", + "61 | \n", + "iris | \n", + "Supervised Classification | \n", + "active | \n", + "10-fold Crossvalidation | \n", + "predictive_accuracy | \n", + "61 | \n", + "class | \n", + "... | \n", + "5 | \n", + "150 | \n", + "0 | \n", + "0 | \n", + "4 | \n", + "1 | \n", + "NaN | \n", + "NaN | \n", + "NaN | \n", + "NaN | \n", + "
| 289 | \n", + "289 | \n", + "1 | \n", + "61 | \n", + "iris | \n", + "Supervised Classification | \n", + "active | \n", + "33% Holdout set | \n", + "predictive_accuracy | \n", + "61 | \n", + "class | \n", + "... | \n", + "5 | \n", + "150 | \n", + "0 | \n", + "0 | \n", + "4 | \n", + "1 | \n", + "NaN | \n", + "NaN | \n", + "NaN | \n", + "NaN | \n", + "
| 1823 | \n", + "1823 | \n", + "1 | \n", + "61 | \n", + "iris | \n", + "Supervised Classification | \n", + "active | \n", + "5 times 2-fold Crossvalidation | \n", + "predictive_accuracy | \n", + "61 | \n", + "class | \n", + "... | \n", + "5 | \n", + "150 | \n", + "0 | \n", + "0 | \n", + "4 | \n", + "1 | \n", + "NaN | \n", + "NaN | \n", + "NaN | \n", + "NaN | \n", + "
| 1939 | \n", + "1939 | \n", + "1 | \n", + "61 | \n", + "iris | \n", + "Supervised Classification | \n", + "active | \n", + "10 times 10-fold Crossvalidation | \n", + "predictive_accuracy | \n", + "61 | \n", + "class | \n", + "... | \n", + "5 | \n", + "150 | \n", + "0 | \n", + "0 | \n", + "4 | \n", + "1 | \n", + "NaN | \n", + "NaN | \n", + "NaN | \n", + "NaN | \n", + "
| 1992 | \n", + "1992 | \n", + "1 | \n", + "61 | \n", + "iris | \n", + "Supervised Classification | \n", + "active | \n", + "Leave one out | \n", + "predictive_accuracy | \n", + "61 | \n", + "class | \n", + "... | \n", + "5 | \n", + "150 | \n", + "0 | \n", + "0 | \n", + "4 | \n", + "1 | \n", + "NaN | \n", + "NaN | \n", + "NaN | \n", + "NaN | \n", + "
5 rows × 24 columns
\n", + "| \n", + " | run_id | \n", + "task_id | \n", + "setup_id | \n", + "flow_id | \n", + "flow_name | \n", + "data_id | \n", + "data_name | \n", + "function | \n", + "upload_time | \n", + "uploader | \n", + "uploader_name | \n", + "value | \n", + "values | \n", + "array_data | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", + "81 | \n", + "59 | \n", + "12 | \n", + "67 | \n", + "weka.BayesNet_K2(1) | \n", + "61 | \n", + "iris | \n", + "predictive_accuracy | \n", + "2014-04-07 00:05:11 | \n", + "1 | \n", + "janvanrijn@gmail.com | \n", + "0.940000 | \n", + "None | \n", + "None | \n", + "
| 1 | \n", + "161 | \n", + "59 | \n", + "13 | \n", + "70 | \n", + "weka.SMO_PolyKernel(1) | \n", + "61 | \n", + "iris | \n", + "predictive_accuracy | \n", + "2014-04-07 00:55:32 | \n", + "1 | \n", + "janvanrijn@gmail.com | \n", + "0.960000 | \n", + "None | \n", + "None | \n", + "
| 2 | \n", + "234 | \n", + "59 | \n", + "1 | \n", + "56 | \n", + "weka.ZeroR(1) | \n", + "61 | \n", + "iris | \n", + "predictive_accuracy | \n", + "2014-04-07 01:33:24 | \n", + "1 | \n", + "janvanrijn@gmail.com | \n", + "0.333333 | \n", + "None | \n", + "None | \n", + "
| 3 | \n", + "447 | \n", + "59 | \n", + "6 | \n", + "61 | \n", + "weka.REPTree(1) | \n", + "61 | \n", + "iris | \n", + "predictive_accuracy | \n", + "2014-04-07 06:26:27 | \n", + "1 | \n", + "janvanrijn@gmail.com | \n", + "0.926667 | \n", + "None | \n", + "None | \n", + "
| 4 | \n", + "473 | \n", + "59 | \n", + "18 | \n", + "77 | \n", + "weka.LogitBoost_DecisionStump(1) | \n", + "61 | \n", + "iris | \n", + "predictive_accuracy | \n", + "2014-04-07 06:39:27 | \n", + "1 | \n", + "janvanrijn@gmail.com | \n", + "0.946667 | \n", + "None | \n", + "None | \n", + "
| \n", + " | run_id | \n", + "task_id | \n", + "setup_id | \n", + "flow_id | \n", + "flow_name | \n", + "data_id | \n", + "data_name | \n", + "function | \n", + "upload_time | \n", + "uploader | \n", + "uploader_name | \n", + "value | \n", + "values | \n", + "array_data | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 144 | \n", + "1849043 | \n", + "59 | \n", + "29015 | \n", + "5500 | \n", + "sklearn.ensemble.forest.RandomForestClassifier... | \n", + "61 | \n", + "iris | \n", + "predictive_accuracy | \n", + "2017-03-03 17:10:12 | \n", + "1 | \n", + "janvanrijn@gmail.com | \n", + "0.946667 | \n", + "None | \n", + "None | \n", + "
| 145 | \n", + "1853409 | \n", + "59 | \n", + "30950 | \n", + "5873 | \n", + "sklearn.pipeline.Pipeline(Imputer=openml.utils... | \n", + "61 | \n", + "iris | \n", + "predictive_accuracy | \n", + "2017-03-21 22:08:01 | \n", + "1 | \n", + "janvanrijn@gmail.com | \n", + "0.960000 | \n", + "None | \n", + "None | \n", + "
| 146 | \n", + "6130126 | \n", + "59 | \n", + "4163633 | \n", + "7108 | \n", + "sklearn.model_selection._search.RandomizedSear... | \n", + "61 | \n", + "iris | \n", + "predictive_accuracy | \n", + "2017-08-21 11:07:40 | \n", + "1 | \n", + "janvanrijn@gmail.com | \n", + "0.960000 | \n", + "None | \n", + "None | \n", + "
| 147 | \n", + "6130128 | \n", + "59 | \n", + "4163634 | \n", + "7108 | \n", + "sklearn.model_selection._search.RandomizedSear... | \n", + "61 | \n", + "iris | \n", + "predictive_accuracy | \n", + "2017-08-21 11:08:06 | \n", + "1 | \n", + "janvanrijn@gmail.com | \n", + "0.946667 | \n", + "None | \n", + "None | \n", + "
| 148 | \n", + "6715383 | \n", + "59 | \n", + "4747289 | \n", + "7117 | \n", + "sklearn.model_selection._search.RandomizedSear... | \n", + "61 | \n", + "iris | \n", + "predictive_accuracy | \n", + "2017-09-01 02:56:44 | \n", + "1 | \n", + "janvanrijn@gmail.com | \n", + "0.960000 | \n", + "None | \n", + "None | \n", + "
| \n", + " | run_id | \n", + "task_id | \n", + "setup_id | \n", + "flow_id | \n", + "flow_name | \n", + "data_id | \n", + "data_name | \n", + "function | \n", + "upload_time | \n", + "uploader | \n", + "uploader_name | \n", + "value | \n", + "values | \n", + "array_data | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 144 | \n", + "1849043 | \n", + "59 | \n", + "29015 | \n", + "5500 | \n", + "sklearn.ensemble.forest.RandomForestClassifier... | \n", + "61 | \n", + "iris | \n", + "predictive_accuracy | \n", + "2017-03-03 17:10:12 | \n", + "1 | \n", + "janvanrijn@gmail.com | \n", + "0.946667 | \n", + "None | \n", + "None | \n", + "
| 145 | \n", + "1853409 | \n", + "59 | \n", + "30950 | \n", + "5873 | \n", + "sklearn.pipeline.Pipeline(Imputer=openml.utils... | \n", + "61 | \n", + "iris | \n", + "predictive_accuracy | \n", + "2017-03-21 22:08:01 | \n", + "1 | \n", + "janvanrijn@gmail.com | \n", + "0.960000 | \n", + "None | \n", + "None | \n", + "
| 146 | \n", + "6130126 | \n", + "59 | \n", + "4163633 | \n", + "7108 | \n", + "sklearn.model_selection._search.RandomizedSear... | \n", + "61 | \n", + "iris | \n", + "predictive_accuracy | \n", + "2017-08-21 11:07:40 | \n", + "1 | \n", + "janvanrijn@gmail.com | \n", + "0.960000 | \n", + "None | \n", + "None | \n", + "
| 147 | \n", + "6130128 | \n", + "59 | \n", + "4163634 | \n", + "7108 | \n", + "sklearn.model_selection._search.RandomizedSear... | \n", + "61 | \n", + "iris | \n", + "predictive_accuracy | \n", + "2017-08-21 11:08:06 | \n", + "1 | \n", + "janvanrijn@gmail.com | \n", + "0.946667 | \n", + "None | \n", + "None | \n", + "
| 190 | \n", + "6946499 | \n", + "59 | \n", + "4978397 | \n", + "7109 | \n", + "sklearn.pipeline.Pipeline(imputation=openmlstu... | \n", + "61 | \n", + "iris | \n", + "predictive_accuracy | \n", + "2017-09-02 22:06:32 | \n", + "1 | \n", + "janvanrijn@gmail.com | \n", + "0.920000 | \n", + "None | \n", + "None | \n", + "
| \n", + " | run_id | \n", + "task_id | \n", + "setup_id | \n", + "flow_id | \n", + "flow_name | \n", + "data_id | \n", + "data_name | \n", + "function | \n", + "upload_time | \n", + "uploader | \n", + "uploader_name | \n", + "value | \n", + "values | \n", + "array_data | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 3549 | \n", + "523926 | \n", + "59 | \n", + "3526 | \n", + "2629 | \n", + "sklearn.ensemble.forest.RandomForestClassifier(8) | \n", + "61 | \n", + "iris | \n", + "predictive_accuracy | \n", + "2016-02-11 22:05:23 | \n", + "869 | \n", + "p.gijsbers@student.tue.nl | \n", + "0.966667 | \n", + "None | \n", + "None | \n", + "
| 4353 | \n", + "8955370 | \n", + "59 | \n", + "6890988 | \n", + "7257 | \n", + "sklearn.ensemble.forest.RandomForestClassifier... | \n", + "61 | \n", + "iris | \n", + "predictive_accuracy | \n", + "2018-04-06 16:32:22 | \n", + "3964 | \n", + "clear.tsai@gmail.com | \n", + "0.960000 | \n", + "None | \n", + "None | \n", + "
| 3587 | \n", + "1852682 | \n", + "59 | \n", + "29263 | \n", + "5500 | \n", + "sklearn.ensemble.forest.RandomForestClassifier... | \n", + "61 | \n", + "iris | \n", + "predictive_accuracy | \n", + "2017-03-15 22:55:18 | \n", + "1022 | \n", + "rso@randalolson.com | \n", + "0.960000 | \n", + "None | \n", + "None | \n", + "
| 4375 | \n", + "8886608 | \n", + "59 | \n", + "6835139 | \n", + "7961 | \n", + "sklearn.pipeline.Pipeline(Imputer=sklearn.prep... | \n", + "61 | \n", + "iris | \n", + "predictive_accuracy | \n", + "2018-03-17 16:46:27 | \n", + "5032 | \n", + "rashmi.kamath01@gmail.com | \n", + "0.960000 | \n", + "None | \n", + "None | \n", + "
| 3107 | \n", + "1843272 | \n", + "59 | \n", + "24071 | \n", + "4830 | \n", + "sklearn.ensemble.forest.RandomForestClassifier... | \n", + "61 | \n", + "iris | \n", + "predictive_accuracy | \n", + "2016-12-08 20:10:03 | \n", + "2 | \n", + "joaquin.vanschoren@gmail.com | \n", + "0.960000 | \n", + "None | \n", + "None | \n", + "
| \n", + " | run_id | \n", + "task_id | \n", + "setup_id | \n", + "flow_id | \n", + "flow_name | \n", + "data_id | \n", + "data_name | \n", + "function | \n", + "upload_time | \n", + "uploader | \n", + "uploader_name | \n", + "value | \n", + "values | \n", + "array_data | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 3630 | \n", + "2039748 | \n", + "59 | \n", + "180922 | \n", + "6048 | \n", + "sklearn.pipeline.Pipeline(dualimputer=helper.d... | \n", + "61 | \n", + "iris | \n", + "predictive_accuracy | \n", + "2017-04-09 01:09:01 | \n", + "1104 | \n", + "jmapvhoof@gmail.com | \n", + "0.986667 | \n", + "None | \n", + "None | \n", + "
| 3631 | \n", + "2039750 | \n", + "59 | \n", + "180924 | \n", + "6048 | \n", + "sklearn.pipeline.Pipeline(dualimputer=helper.d... | \n", + "61 | \n", + "iris | \n", + "predictive_accuracy | \n", + "2017-04-09 01:17:39 | \n", + "1104 | \n", + "jmapvhoof@gmail.com | \n", + "0.986667 | \n", + "None | \n", + "None | \n", + "
| 3624 | \n", + "2012939 | \n", + "59 | \n", + "157622 | \n", + "6048 | \n", + "sklearn.pipeline.Pipeline(dualimputer=helper.d... | \n", + "61 | \n", + "iris | \n", + "predictive_accuracy | \n", + "2017-04-06 23:29:28 | \n", + "1104 | \n", + "jmapvhoof@gmail.com | \n", + "0.986667 | \n", + "None | \n", + "None | \n", + "
| 3618 | \n", + "2012930 | \n", + "59 | \n", + "157613 | \n", + "6048 | \n", + "sklearn.pipeline.Pipeline(dualimputer=helper.d... | \n", + "61 | \n", + "iris | \n", + "predictive_accuracy | \n", + "2017-04-06 23:00:24 | \n", + "1104 | \n", + "jmapvhoof@gmail.com | \n", + "0.986667 | \n", + "None | \n", + "None | \n", + "
| 3626 | \n", + "2012941 | \n", + "59 | \n", + "157624 | \n", + "6048 | \n", + "sklearn.pipeline.Pipeline(dualimputer=helper.d... | \n", + "61 | \n", + "iris | \n", + "predictive_accuracy | \n", + "2017-04-07 01:36:00 | \n", + "1104 | \n", + "jmapvhoof@gmail.com | \n", + "0.986667 | \n", + "None | \n", + "None | \n", + "