{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3b466d92",
   "metadata": {},
   "source": [
    "# Distinguish Failed Experiments\n",
    "\n",
    "Modeling runs can fail for a number of reasons. When logging with rubicon_ml, a failed run may result in an empty or incomplete experiment. In this example, we'll walk through how to handle such experiments."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4f58990",
   "metadata": {},
   "source": [
    "First lets simulate the problem. To do this we'll create an estimator that will fail on it's `fit()` 30% of the time.  We will consider any pipeline that has a learned attribute `self.state_` to have \"succeeded,\" and any that does not to have \"failed.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "f9a506c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.base import BaseEstimator\n",
    "import random\n",
    "\n",
    "class BadEstimator(BaseEstimator):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.knn = KNeighborsClassifier(n_neighbors=2)\n",
    "    def fit(self, X, y):\n",
    "        self.knn.fit(X, y)\n",
    "        output=random.random()\n",
    "        if output>.3:\n",
    "            self.state_=output\n",
    "    def score(self, X):\n",
    "        knn_score = self.knn.score(X)\n",
    "        return knn_score"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d62eaa1",
   "metadata": {},
   "source": [
    "Next, lets create a `rubicon_ml` project to log our experimenation to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "7ccb023c",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from rubicon_ml.sklearn import make_pipeline\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.impute import SimpleImputer\n",
    "from rubicon_ml import Rubicon\n",
    "\n",
    "random.seed(17)\n",
    "\n",
    "rubicon = Rubicon(\n",
    "    persistence=\"memory\",\n",
    ")\n",
    "project = rubicon.get_or_create_project(name=\"Failed Experiments\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "852bd664",
   "metadata": {},
   "source": [
    "Now let's create a `rubicon_ml.sklearn` pipeline with this sporadically failing estimator and attempt to `fit` the pipeline twenty times.  Tag any experiment that doesn't have a valid `state_` attribute for failures with `exp.add_tags([\"failed\"])` and passed experiments with `exp.add_tags([\"passed\"])`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "379dc1be",
   "metadata": {},
   "outputs": [],
   "source": [
    "X = [[1], [1], [1], [1]]\n",
    "y = [1, 1, 1, 1]\n",
    "for _ in range(20):\n",
    "    pipe=make_pipeline(project, SimpleImputer(strategy=\"mean\"),BadEstimator())\n",
    "    pipe.fit(X,y)\n",
    "    if not hasattr(pipe[\"badestimator\"],\"state_\"):\n",
    "         pipe.experiment.add_tags([\"failed\"])\n",
    "    else:\n",
    "         pipe.experiment.add_tags([\"passed\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e03bd0a",
   "metadata": {},
   "source": [
    "Finally, we can now retrieve all our failed experiments by passing the `tags=[\"failed\"]` to `project.experiments()`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "1ef73ac0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Experiment(project_name='Failed Experiments', id='375a55ec-9e50-4c59-86c8-e06be471d45e', name='RubiconPipeline experiment', description=None, model_name=None, branch_name=None, commit_hash=None, training_metadata=None, tags=['failed'], created_at=datetime.datetime(2022, 5, 10, 14, 50, 44, 669757))\n",
      "Experiment(project_name='Failed Experiments', id='fce82fb6-58d8-42df-a40b-304bc83826b5', name='RubiconPipeline experiment', description=None, model_name=None, branch_name=None, commit_hash=None, training_metadata=None, tags=['failed'], created_at=datetime.datetime(2022, 5, 10, 14, 50, 44, 676902))\n",
      "Experiment(project_name='Failed Experiments', id='912b9efe-db1f-4ff2-b7c3-51d23bc60acf', name='RubiconPipeline experiment', description=None, model_name=None, branch_name=None, commit_hash=None, training_metadata=None, tags=['failed'], created_at=datetime.datetime(2022, 5, 10, 14, 50, 44, 678565))\n",
      "Experiment(project_name='Failed Experiments', id='75f4d429-b67e-4d16-a634-700b600224fc', name='RubiconPipeline experiment', description=None, model_name=None, branch_name=None, commit_hash=None, training_metadata=None, tags=['failed'], created_at=datetime.datetime(2022, 5, 10, 14, 50, 44, 683442))\n",
      "Experiment(project_name='Failed Experiments', id='e0ca4d92-5c37-4118-bfc8-96253fe390c9', name='RubiconPipeline experiment', description=None, model_name=None, branch_name=None, commit_hash=None, training_metadata=None, tags=['failed'], created_at=datetime.datetime(2022, 5, 10, 14, 50, 44, 697318))\n"
     ]
    }
   ],
   "source": [
    "for exp in project.experiments(tags=[\"failed\"]):\n",
    "    print(exp)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bed451a5",
   "metadata": {},
   "source": [
    "We can also see that the pipeline passed ~70% of the time and ~30% of the time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "d61e7839",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(project.experiments(tags=[\"failed\"]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "06789044",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "15"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(project.experiments(tags=[\"passed\"]))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}