View this notebook on GitHub or run it yourself on Binder!

Integrate with Scikit-learn#

This example shows how you can integrate rubicon_ml into your Scikit-learn pipelines to enable automatic logging of parameters and metrics as you fit and score your models!

Simple pipeline run#

Using the RubiconPipeline class, you can set up a enhanced Scikit-learn pipeline with automated logging.

from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline

from rubicon_ml import Rubicon
from rubicon_ml.sklearn import RubiconPipeline

rubicon = Rubicon(persistence="memory")
project = rubicon.get_or_create_project("Rubicon Pipeline Example")

X, y = make_classification(random_state=0)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

pipe = RubiconPipeline(
    [('scaler', StandardScaler()), ('svc', SVC())],
RubiconPipeline(project=<rubicon_ml.client.project.Project object at 0x10eab9570>,
                steps=[('scaler', StandardScaler()), ('svc', SVC())])
[2]:, y_train)
RubiconPipeline(project=<rubicon_ml.client.project.Project object at 0x10eab9570>,
                steps=[('scaler', StandardScaler()), ('svc', SVC())])
pipe.score(X_test, y_test)

During the pipeline run, an experiment was automatically created and the corresponding parameters and metrics logged to it. Afterwards, you can use the rubicon_ml library to pull these experiments back or view them by running the dashboard.

for experiment in project.experiments():
    parameters = [(, p.value) for p in experiment.parameters()]
    metrics = [(, m.value) for m in experiment.metrics()]

        f"experiment {}\n"
        f"parameters: {parameters}\nmetrics: {metrics}"
experiment 246c4f1f-5302-4e2e-bcc4-cd28f157c7b9
parameters: [('scaler__copy', True), ('scaler__with_mean', True), ('scaler__with_std', True), ('svc__C', 1.0), ('svc__break_ties', False), ('svc__cache_size', 200), ('svc__class_weight', None), ('svc__coef0', 0.0), ('svc__decision_function_shape', 'ovr'), ('svc__degree', 3), ('svc__gamma', 'scale'), ('svc__kernel', 'rbf'), ('svc__max_iter', -1), ('svc__probability', False), ('svc__random_state', None), ('svc__shrinking', True), ('svc__tol', 0.001), ('svc__verbose', False)]
metrics: [('score', 0.88)]

By default, rubicon_ml’s logging is very verbose. It captures each parameter passed to each stage of your pipeline. In the next example we’ll see how to target our logging a bit more.

A more realistic example using GridSearch#

GridSearch is commonly used to test many different parameters across an estimator or pipeline in the hopes of finding the optimal parameter set. The RubiconPipeline fits the Scikit-learn estimator specificaion, so it can be passed to Scikit-learn’s GridSearchCV to automatically log each set of parameters tried in the grid search to an individual experiment. Then, all of these experiments can be explored with the dashboard!

This example is adapted from this Scikit-learn example.

from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.linear_model import SGDClassifier
from sklearn.model_selection import GridSearchCV

categories = ["alt.atheism", "talk.religion.misc"]
data = fetch_20newsgroups(subset='train', categories=categories)

You can pass user-defined loggers to the RubiconPipeline to have more control over exactly which parameters are logged for specific estimators. For example, you can use the FilteredLogger class to select or ignore parameters on any estimator.

import os

from rubicon_ml import Rubicon
from rubicon_ml.sklearn import FilterEstimatorLogger, RubiconPipeline

root_dir = os.environ.get("RUBICON_ROOT", "rubicon-root")
root_path = f"{os.path.dirname(os.getcwd())}/{root_dir}"

rubicon = Rubicon(persistence="filesystem", root_dir=root_path)
project = rubicon.get_or_create_project("Grid Search")

pipeline = RubiconPipeline(
        ("vect", CountVectorizer()),
        ("tfidf", TfidfTransformer()),
        ("clf", SGDClassifier()),
    user_defined_loggers = {
        "vect": FilterEstimatorLogger(select=["max_df"]),
        "tfidf": FilterEstimatorLogger(ignore_all=True),
        "clf": FilterEstimatorLogger(select=["max_iter", "alpha", "penalty"]),
        "name": "logged from a RubiconPipeline",
        "model_name": SGDClassifier.__name__,

Let’s define a parameter grid and run some experiments!

parameters = {
    "vect__max_df": (0.5, 0.75, 1.0),
    "vect__ngram_range": ((1, 1), (1, 2)),
    "clf__max_iter": (10, 20),
    "clf__alpha": (0.00001, 0.000001),
    "clf__penalty": ("l2", "elasticnet"),

grid_search = GridSearchCV(pipeline, parameters, cv=2, n_jobs=-1, refit=False),
             estimator=RubiconPipeline(experiment_kwargs={'model_name': 'SGDClassifier',
                                                          'name': 'logged from '
                                                                  'a '
                                       project=<rubicon_ml.client.project.Project object at 0x1a7aeb670>,
                                       steps=[('vect', CountVectorizer()),
                                              ('tfidf', TfidfTransformer()),
                                              ('clf', SGDClassifier())],
                                       user_defined_loggers={'clf': <rubicon_ml.sklearn.filter_estimator_l...
                                                             'tfidf': <rubicon_ml.sklearn.filter_estimator_logger.FilterEstimatorLogger object at 0x1a7b59f30>,
                                                             'vect': <rubicon_ml.sklearn.filter_estimator_logger.FilterEstimatorLogger object at 0x1a7b588b0>}),
             param_grid={'clf__alpha': (1e-05, 1e-06),
                         'clf__max_iter': (10, 20),
                         'clf__penalty': ('l2', 'elasticnet'),
                         'vect__max_df': (0.5, 0.75, 1.0),
                         'vect__ngram_range': ((1, 1), (1, 2))},

Fetching the best parameters from the GridSearchCV object involves digging into the object’s properties and doesn’t easily paint a full picture of our of the experimentation.

print(f"Best score: {grid_search.best_score_}")
full_results = grid_search.cv_results_
Best score: 0.9334874626930701

With rubicon_ml’s dashboard, we can view all of the experiments and easily compare them!

from rubicon_ml.viz import Dashboard

Dash is running on

 * Serving Flask app 'rubicon_ml.viz.base' (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off