View this notebook on GitHub or run it yourself on Binder!

---

Logging Concurrently

Let’s see how a few other popular scikit-learn models perform with the wine dataset. rubicon_ml’s filesystem logging is totally thread-safe, so we can test a lot of model configurations at once.

Note: rubicon_ml’s in-memory logging is inherently not threadsafe as each thread has its own memory.

import os

from rubicon_ml import Rubicon


root_dir = os.environ.get("RUBICON_ROOT", "rubicon-root")
root_path = f"{os.path.dirname(os.getcwd())}/{root_dir}"

rubicon = Rubicon(persistence="filesystem", root_dir=root_path)
project = rubicon.get_or_create_project(
    "Concurrent Experiments",
    description="training multiple models in parallel",
)

project

<rubicon_ml.client.project.Project at 0x155660d00>

For a recap of the contents of the wine dataset, check out wine.DESCR and wine.data. We’ll put together a training dataset using a subset of the data.

from sklearn.datasets import load_wine
from sklearn.model_selection import train_test_split


wine = load_wine()
wine_feature_names = wine.feature_names
wine_datasets = train_test_split(
    wine["data"],
    wine["target"],
    test_size=0.25,
)

We’ll use this run_experiment function to log a new experiment to the provided project then train, run and log a model of type classifier_cls using the training and testing data in wine_datasets.

from collections import namedtuple


SklearnTrainingMetadata = namedtuple("SklearnTrainingMetadata", "module_name method")

def run_experiment(project, classifier_cls, wine_datasets, feature_names, **kwargs):
    X_train, X_test, y_train, y_test = wine_datasets

    experiment = project.log_experiment(
        training_metadata=[
            SklearnTrainingMetadata("sklearn.datasets", "load_wine"),
        ],
        model_name=classifier_cls.__name__,
        tags=[classifier_cls.__name__],
    )

    for key, value in kwargs.items():
        experiment.log_parameter(key, value)

    for name in feature_names:
        experiment.log_feature(name)

    classifier = classifier_cls(**kwargs)
    classifier.fit(X_train, y_train)
    classifier.predict(X_test)

    accuracy = classifier.score(X_test, y_test)

    experiment.log_metric("accuracy", accuracy)

    if accuracy >= .95:
        experiment.add_tags(["success"])
    else:
        experiment.add_tags(["failure"])

If you’re familiar with trying to use the multiprocessing library in iPython 3, you’ll know the two don’t play along well. In order for the multiprocessing library to locate the run_experiment function, we’ll need to import it from a separate file. ./logging_concurrently.py defines the same run_experiment function as above.

from logging_concurrently import run_experiment

This time we’ll take a look at three classifiers - RandomForestClassifier, DecisionTreeClassifier, and KNeighborsClassifier - to see which performs best. Each classifier will be run across four sets of parameters (provided as kwargs to run_experiment), for a total of 12 experiments. Here, we’ll build up a list of processes that will run each experiment in parallel.

import multiprocessing

from sklearn.ensemble import RandomForestClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier


processes = []

for n_estimators in [10, 20, 30, 40]:
    processes.append(multiprocessing.Process(
        target=run_experiment,
        args=[project, RandomForestClassifier, wine_datasets, wine_feature_names],
        kwargs={"n_estimators": n_estimators},
    ))

for n_neighbors in [5, 10, 15, 20]:
    processes.append(multiprocessing.Process(
        target=run_experiment,
        args=[project, KNeighborsClassifier, wine_datasets, wine_feature_names],
        kwargs={"n_neighbors": n_neighbors},
    ))

for criterion in ["gini", "entropy"]:
    for splitter in ["best", "random"]:
        processes.append(multiprocessing.Process(
            target=run_experiment,
            args=[project, DecisionTreeClassifier, wine_datasets, wine_feature_names],
            kwargs={
                "criterion": criterion,
                "splitter": splitter,
            },
        ))

Let’s run all our experiments in parallel!

for process in processes:
    process.start()

for process in processes:
    process.join()

Now we can validate that we successfully logged all 12 experiments to our project.

len(project.experiments())

12

Let’s see which experiments we tagged as successful and what type of model they used.

for e in project.experiments(tags=["success"]):
    print(f"experiment {e.id[:8]} was successful using a {e.model_name}")

experiment 3a144831 was successful using a RandomForestClassifier
experiment 67961ac9 was successful using a RandomForestClassifier
experiment 712564c0 was successful using a RandomForestClassifier
experiment f143ea43 was successful using a RandomForestClassifier

We can also take a deeper look at any of our experiments.

first_experiment = project.experiments()[0]

training_metadata = SklearnTrainingMetadata(*first_experiment.training_metadata)
tags = first_experiment.tags

parameters = [(p.name, p.value) for p in first_experiment.parameters()]
metrics = [(m.name, m.value) for m in first_experiment.metrics()]

print(
    f"experiment {first_experiment.id}\n"
    f"training metadata: {training_metadata}\ntags: {tags}\n"
    f"parameters: {parameters}\nmetrics: {metrics}"
)

experiment 0b6b8114-fe72-49af-813b-23e1667c1486
training metadata: SklearnTrainingMetadata(module_name='sklearn.datasets', method='load_wine')
tags: ['KNeighborsClassifier', 'failure']
parameters: [('n_neighbors', 10)]
metrics: [('accuracy', 0.7111111111111111)]

Or we could grab the project’s data as a dataframe!

ddf = rubicon.get_project_as_df("Concurrent Experiments", df_type="dask")
ddf.compute()

	id	name	description	model_name	commit_hash	tags	created_at	splitter	criterion	n_estimators	n_neighbors	accuracy
0	aa4faa9b-3051-454d-915d-54b5c16f6e34	None	None	DecisionTreeClassifier	None	[DecisionTreeClassifier, failure]	2021-04-28 12:45:27.759817	best	gini	NaN	NaN	0.933333
1	7aec1e7b-c7dd-43e7-b20c-a400b2141d23	None	None	DecisionTreeClassifier	None	[DecisionTreeClassifier, failure]	2021-04-28 12:45:27.752157	best	entropy	NaN	NaN	0.933333
2	3a144831-b093-4dc3-85a6-71bc7a4e26a4	None	None	RandomForestClassifier	None	[RandomForestClassifier, success]	2021-04-28 12:45:27.715676	NaN	NaN	20.0	NaN	1.000000
3	7503bcf0-2daa-4c66-9b45-a75863d446a5	None	None	DecisionTreeClassifier	None	[DecisionTreeClassifier, failure]	2021-04-28 12:45:27.714952	random	gini	NaN	NaN	0.911111
4	f143ea43-d780-4639-a082-5c81b850f1e0	None	None	RandomForestClassifier	None	[RandomForestClassifier, success]	2021-04-28 12:45:27.636078	NaN	NaN	10.0	NaN	0.977778
5	67961ac9-4bd5-4916-925c-fd38844d78c6	None	None	RandomForestClassifier	None	[RandomForestClassifier, success]	2021-04-28 12:45:27.635968	NaN	NaN	30.0	NaN	0.977778
6	8b6b9b9a-1e06-4e0a-af2e-9649d79872e3	None	None	KNeighborsClassifier	None	[KNeighborsClassifier, failure]	2021-04-28 12:45:27.619764	NaN	NaN	NaN	15.0	0.733333
7	3c679288-1402-4ffb-bbb1-23fcfd0b1415	None	None	DecisionTreeClassifier	None	[DecisionTreeClassifier, failure]	2021-04-28 12:45:27.615204	random	entropy	NaN	NaN	0.888889
8	38aa6e7b-efcc-4ee5-9950-5449ef8681ff	None	None	KNeighborsClassifier	None	[KNeighborsClassifier, failure]	2021-04-28 12:45:27.585590	NaN	NaN	NaN	20.0	0.711111
9	0b6b8114-fe72-49af-813b-23e1667c1486	None	None	KNeighborsClassifier	None	[KNeighborsClassifier, failure]	2021-04-28 12:45:27.583647	NaN	NaN	NaN	10.0	0.711111
10	a0b7e295-6047-45da-81e6-08dc8288a2eb	None	None	KNeighborsClassifier	None	[KNeighborsClassifier, failure]	2021-04-28 12:45:27.564990	NaN	NaN	NaN	5.0	0.733333
11	712564c0-e113-42ea-b59c-90a5ce0cf37b	None	None	RandomForestClassifier	None	[RandomForestClassifier, success]	2021-04-28 12:45:27.511807	NaN	NaN	40.0	NaN	0.955556