View this notebook on GitHub or run it yourself on Binder!

Manage Experiment Relationships#

rubicon-ml experiments can be tagged with special identifiers to denote a parent/child relationship. This can be used to track hierarchical or iterative experiments, among other things.

First, let’s create a project.

[1]:

from rubicon_ml import Rubicon

rubicon = Rubicon(persistence="memory")
project = rubicon.create_project(name="hierarchical experiments")
project

[1]:

<rubicon_ml.client.project.Project at 0x121d7af50>

Hierarchical experiments#

Now we can log some experiments in a nested loop. Imagine logging an experiment for each node of a gradient boosted tree, or something along those lines.

We can use parent_experiment.add_child_experiment(child_experiment) to automatically add tags to both parent_experiment and child_experiment that represent their relationship.

[2]:

root_experiment = project.log_experiment(name="root")

for n in range(3):
    node_experiment = project.log_experiment(name=f"node_{n}")
    root_experiment.add_child_experiment(node_experiment)

    for m in range(2):
        nested_node_experiment = project.log_experiment(name=f"node_{n}_{m}")
        node_experiment.add_child_experiment(nested_node_experiment)

To retrieve experiments, start at the root experiment and call get_child_experiments to return a list of rubicon-ml objects representing each of the tagged child experiments.

[3]:

for experiment in root_experiment.get_child_experiments():
    print("id:", experiment.id)
    print("tags:", [t for t in experiment.tags if "child" in t], "\n")

    for nested_experiment in experiment.get_child_experiments():
        print("\tid:", nested_experiment.id)
        print("\ttags:", nested_experiment.tags, "\n")

id: f8c897d7-852d-4020-8715-264452a5b8ab
tags: ['child:754dff35-aa87-4385-9bcd-af5c1f5b0b7e', 'child:fdf25d72-d1bb-47bc-8cb2-4a088ed0ba33']

        id: 754dff35-aa87-4385-9bcd-af5c1f5b0b7e
        tags: ['parent:f8c897d7-852d-4020-8715-264452a5b8ab']

        id: fdf25d72-d1bb-47bc-8cb2-4a088ed0ba33
        tags: ['parent:f8c897d7-852d-4020-8715-264452a5b8ab']

id: 5f8c14c1-50d9-4c49-b0b2-6a59c6f3d707
tags: ['child:d9012c99-2888-43d1-833b-78d51de75a3a', 'child:c65f8c5e-bf5b-4a82-94cb-8b669545b951']

        id: d9012c99-2888-43d1-833b-78d51de75a3a
        tags: ['parent:5f8c14c1-50d9-4c49-b0b2-6a59c6f3d707']

        id: c65f8c5e-bf5b-4a82-94cb-8b669545b951
        tags: ['parent:5f8c14c1-50d9-4c49-b0b2-6a59c6f3d707']

id: da33e918-96c0-4e56-9075-7941515cc18f
tags: ['child:7c5b2f4b-1e7c-40be-8cda-8f3a00067e98', 'child:cd30f3b2-bd63-4318-974a-6668648bf4ac']

        id: 7c5b2f4b-1e7c-40be-8cda-8f3a00067e98
        tags: ['parent:da33e918-96c0-4e56-9075-7941515cc18f']

        id: cd30f3b2-bd63-4318-974a-6668648bf4ac
        tags: ['parent:da33e918-96c0-4e56-9075-7941515cc18f']

Iterative experiments#

We can leverage add_child_experiment to maintain iterative relationships too. This could be used to log metadata about of each iteration of recursive feature elimination and preserve the linear history of the model training.

[4]:

current_experiment = project.log_experiment(name="experiment_0")

for n in range(3):
    next_experiment = project.log_experiment(name=f"experiment_{n+1}")
    current_experiment.add_child_experiment(next_experiment)

    current_experiment = next_experiment

last_experiment = current_experiment

Similarly to get_child_experiments, we can use get_parent_experiment to return a rubicon-ml object representing the tagged parent experiment.

[5]:

experiments = [last_experiment]

while len(experiments) != 0:
    experiment = experiments[0]

    print("name:", experiment.name)
    print("\tid:", experiment.id)
    print("\ttags:", experiment.tags, "\n")

    experiments = experiment.get_parent_experiments()

name: experiment_3
        id: a85ba9d9-1473-44c2-b3e9-8a744534485b
        tags: ['parent:47ecf8a7-b799-4308-994b-aa6d8698dc2b']

name: experiment_2
        id: 47ecf8a7-b799-4308-994b-aa6d8698dc2b
        tags: ['child:a85ba9d9-1473-44c2-b3e9-8a744534485b', 'parent:aea5f005-9792-442c-b98c-8d9b9e39f99b']

name: experiment_1
        id: aea5f005-9792-442c-b98c-8d9b9e39f99b
        tags: ['child:47ecf8a7-b799-4308-994b-aa6d8698dc2b', 'parent:f4a393ef-0b32-4f70-ac82-07a0877da328']

name: experiment_0
        id: f4a393ef-0b32-4f70-ac82-07a0877da328
        tags: ['child:aea5f005-9792-442c-b98c-8d9b9e39f99b']