{ "cells": [ { "cell_type": "markdown", "id": "815af48c-67f4-4335-bf16-11068f7094bb", "metadata": {}, "source": [ "# Manage Experiment Relationships\n", "\n", "``rubicon-ml`` experiments can be tagged with special identifiers to denote a parent/child relationship.\n", "This can be used to track hierarchical or iterative experiments, among other things.\n", "\n", "First, let's create a project." ] }, { "cell_type": "code", "execution_count": 1, "id": "c44f140a-0d40-4919-b058-8f986dd9bcb1", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from rubicon_ml import Rubicon\n", "\n", "rubicon = Rubicon(persistence=\"memory\")\n", "project = rubicon.create_project(name=\"hierarchical experiments\")\n", "project" ] }, { "cell_type": "markdown", "id": "d9aee6c2-8891-4a5b-98d6-37cce80bb40f", "metadata": {}, "source": [ "## Hierarchical experiments\n", "\n", "Now we can log some experiments in a nested loop. Imagine logging an experiment for each node of a\n", "gradient boosted tree, or something along those lines.\n", "\n", "We can use ``parent_experiment.add_child_experiment(child_experiment)`` to automatically add tags\n", "to both ``parent_experiment`` and ``child_experiment`` that represent their relationship." ] }, { "cell_type": "code", "execution_count": 2, "id": "af1bd79b-dd77-4ccb-affb-8abea69f581b", "metadata": {}, "outputs": [], "source": [ "root_experiment = project.log_experiment(name=\"root\")\n", "\n", "for n in range(3):\n", " node_experiment = project.log_experiment(name=f\"node_{n}\")\n", " root_experiment.add_child_experiment(node_experiment)\n", "\n", " for m in range(2):\n", " nested_node_experiment = project.log_experiment(name=f\"node_{n}_{m}\")\n", " node_experiment.add_child_experiment(nested_node_experiment)" ] }, { "cell_type": "markdown", "id": "a2f6583c-081a-4afe-ba3a-5e6b8744f274", "metadata": {}, "source": [ "To retrieve experiments, start at the root experiment and call ``get_child_experiments`` to return a\n", "list of ``rubicon-ml`` objects representing each of the tagged child experiments." ] }, { "cell_type": "code", "execution_count": 3, "id": "30068979-44ad-4fd5-9dab-6a2dbee66078", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "id: f8c897d7-852d-4020-8715-264452a5b8ab\n", "tags: ['child:754dff35-aa87-4385-9bcd-af5c1f5b0b7e', 'child:fdf25d72-d1bb-47bc-8cb2-4a088ed0ba33'] \n", "\n", "\tid: 754dff35-aa87-4385-9bcd-af5c1f5b0b7e\n", "\ttags: ['parent:f8c897d7-852d-4020-8715-264452a5b8ab'] \n", "\n", "\tid: fdf25d72-d1bb-47bc-8cb2-4a088ed0ba33\n", "\ttags: ['parent:f8c897d7-852d-4020-8715-264452a5b8ab'] \n", "\n", "id: 5f8c14c1-50d9-4c49-b0b2-6a59c6f3d707\n", "tags: ['child:d9012c99-2888-43d1-833b-78d51de75a3a', 'child:c65f8c5e-bf5b-4a82-94cb-8b669545b951'] \n", "\n", "\tid: d9012c99-2888-43d1-833b-78d51de75a3a\n", "\ttags: ['parent:5f8c14c1-50d9-4c49-b0b2-6a59c6f3d707'] \n", "\n", "\tid: c65f8c5e-bf5b-4a82-94cb-8b669545b951\n", "\ttags: ['parent:5f8c14c1-50d9-4c49-b0b2-6a59c6f3d707'] \n", "\n", "id: da33e918-96c0-4e56-9075-7941515cc18f\n", "tags: ['child:7c5b2f4b-1e7c-40be-8cda-8f3a00067e98', 'child:cd30f3b2-bd63-4318-974a-6668648bf4ac'] \n", "\n", "\tid: 7c5b2f4b-1e7c-40be-8cda-8f3a00067e98\n", "\ttags: ['parent:da33e918-96c0-4e56-9075-7941515cc18f'] \n", "\n", "\tid: cd30f3b2-bd63-4318-974a-6668648bf4ac\n", "\ttags: ['parent:da33e918-96c0-4e56-9075-7941515cc18f'] \n", "\n" ] } ], "source": [ "for experiment in root_experiment.get_child_experiments():\n", " print(\"id:\", experiment.id)\n", " print(\"tags:\", [t for t in experiment.tags if \"child\" in t], \"\\n\")\n", "\n", " for nested_experiment in experiment.get_child_experiments():\n", " print(\"\\tid:\", nested_experiment.id)\n", " print(\"\\ttags:\", nested_experiment.tags, \"\\n\")" ] }, { "cell_type": "markdown", "id": "12bc18eb-0edd-4054-840b-c4b969a150fb", "metadata": {}, "source": [ "## Iterative experiments\n", "\n", "We can leverage ``add_child_experiment`` to maintain iterative relationships too. This could be\n", "used to log metadata about of each iteration of recursive feature elimination and preserve the\n", "linear history of the model training." ] }, { "cell_type": "code", "execution_count": 4, "id": "0f104792-7cbf-4508-b37c-11dfb158b608", "metadata": {}, "outputs": [], "source": [ "current_experiment = project.log_experiment(name=\"experiment_0\")\n", "\n", "for n in range(3):\n", " next_experiment = project.log_experiment(name=f\"experiment_{n+1}\")\n", " current_experiment.add_child_experiment(next_experiment)\n", "\n", " current_experiment = next_experiment\n", "\n", "last_experiment = current_experiment" ] }, { "cell_type": "markdown", "id": "a0f42e74-8fc2-460a-9282-ed0492639d75", "metadata": {}, "source": [ "Similarly to ``get_child_experiments``, we can use ``get_parent_experiment`` to return a ``rubicon-ml``\n", "object representing the tagged parent experiment." ] }, { "cell_type": "code", "execution_count": 5, "id": "35613e52-84f1-4d2e-8e3c-8c0f2b731d89", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "name: experiment_3\n", "\tid: a85ba9d9-1473-44c2-b3e9-8a744534485b\n", "\ttags: ['parent:47ecf8a7-b799-4308-994b-aa6d8698dc2b'] \n", "\n", "name: experiment_2\n", "\tid: 47ecf8a7-b799-4308-994b-aa6d8698dc2b\n", "\ttags: ['child:a85ba9d9-1473-44c2-b3e9-8a744534485b', 'parent:aea5f005-9792-442c-b98c-8d9b9e39f99b'] \n", "\n", "name: experiment_1\n", "\tid: aea5f005-9792-442c-b98c-8d9b9e39f99b\n", "\ttags: ['child:47ecf8a7-b799-4308-994b-aa6d8698dc2b', 'parent:f4a393ef-0b32-4f70-ac82-07a0877da328'] \n", "\n", "name: experiment_0\n", "\tid: f4a393ef-0b32-4f70-ac82-07a0877da328\n", "\ttags: ['child:aea5f005-9792-442c-b98c-8d9b9e39f99b'] \n", "\n" ] } ], "source": [ "experiments = [last_experiment]\n", "\n", "while len(experiments) != 0:\n", " experiment = experiments[0]\n", "\n", " print(\"name:\", experiment.name)\n", " print(\"\\tid:\", experiment.id)\n", " print(\"\\ttags:\", experiment.tags, \"\\n\")\n", "\n", " experiments = experiment.get_parent_experiments()" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.6" } }, "nbformat": 4, "nbformat_minor": 5 }