View this notebook on GitHub or run it yourself on Binder!


Ignoring Exceptions with Failure Modes

rubicon-ml is often used for logging in scenarios that require high availability, like model inference pipelines running in production environments. If something were to go wrong with rubicon-ml during live model inference, we could end up halting predictions just for a logging issue. rubicon-ml’s configurable failure modes allow users to choose what to do with rubicon-ml exceptions!

First, let’s try to get a project that we haven’t yet created. This will show the default failure behavior - raising a RubiconException that halts execution of the code it originated from.

[1]:
from rubicon_ml import Rubicon


rb = Rubicon(persistence="memory")
rb.get_project(name="failure modes")
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
File ~/mambaforge/envs/rubicon-ml-dev/lib/python3.10/site-packages/fsspec/implementations/memory.py:213, in MemoryFileSystem.cat_file(self, path, start, end, **kwargs)
    212 try:
--> 213     return bytes(self.store[path].getbuffer()[start:end])
    214 except KeyError:

KeyError: '/root/failure-modes/metadata.json'

During handling of the above exception, another exception occurred:

FileNotFoundError                         Traceback (most recent call last)
File ~/github/capitalone/rubicon-ml/rubicon_ml/repository/base.py:102, in BaseRepository.get_project(self, project_name)
    101 try:
--> 102     project = json.loads(self.filesystem.cat(project_metadata_path))
    103 except FileNotFoundError:

File ~/mambaforge/envs/rubicon-ml-dev/lib/python3.10/site-packages/fsspec/spec.py:755, in AbstractFileSystem.cat(self, path, recursive, on_error, **kwargs)
    754 else:
--> 755     return self.cat_file(paths[0], **kwargs)

File ~/mambaforge/envs/rubicon-ml-dev/lib/python3.10/site-packages/fsspec/implementations/memory.py:215, in MemoryFileSystem.cat_file(self, path, start, end, **kwargs)
    214 except KeyError:
--> 215     raise FileNotFoundError(path)

FileNotFoundError: /root/failure-modes/metadata.json

During handling of the above exception, another exception occurred:

RubiconException                          Traceback (most recent call last)
Cell In [1], line 5
      1 from rubicon_ml import Rubicon
      4 rb = Rubicon(persistence="memory")
----> 5 rb.get_project(name="failure modes")

File ~/github/capitalone/rubicon-ml/rubicon_ml/client/utils/exception_handling.py:46, in failsafe.<locals>.wrapper(*args, **kwargs)
     44 except Exception as e:
     45     if FAILURE_MODE == "raise":
---> 46         raise e
     47     elif FAILURE_MODE == "warn":
     48         warnings.warn(traceback.format_exc(limit=TRACEBACK_LIMIT, chain=TRACEBACK_CHAIN))

File ~/github/capitalone/rubicon-ml/rubicon_ml/client/utils/exception_handling.py:43, in failsafe.<locals>.wrapper(*args, **kwargs)
     41 def wrapper(*args, **kwargs):
     42     try:
---> 43         return func(*args, **kwargs)
     44     except Exception as e:
     45         if FAILURE_MODE == "raise":

File ~/github/capitalone/rubicon-ml/rubicon_ml/client/rubicon.py:123, in Rubicon.get_project(self, name, id)
    120     raise ValueError("`name` OR `id` required.")
    122 if name is not None:
--> 123     project = self.repository.get_project(name)
    124     project = Project(project, self.config)
    125 else:

File ~/github/capitalone/rubicon-ml/rubicon_ml/repository/base.py:104, in BaseRepository.get_project(self, project_name)
    102     project = json.loads(self.filesystem.cat(project_metadata_path))
    103 except FileNotFoundError:
--> 104     raise RubiconException(f"No project with name '{project_name}' found.")
    106 return domain.Project(**project)

RubiconException: No project with name 'failure modes' found.

But, let’s say we’re far more concerned with keeping our code running than we are with whether or not our logs get logged.

We can set_failure_mode to “warn” to instead raise warnings (via the builtin warnings.warn) whenever rubicon-ml encounters an exception and continue execution of the offending code.

[2]:
from rubicon_ml import set_failure_mode


set_failure_mode("warn")

rb.get_project(name="failure modes")
/Users/nvd215/github/capitalone/rubicon-ml/rubicon_ml/client/utils/exception_handling.py:48: UserWarning: Traceback (most recent call last):
  File "/Users/nvd215/github/capitalone/rubicon-ml/rubicon_ml/client/utils/exception_handling.py", line 43, in wrapper
    return func(*args, **kwargs)
  File "/Users/nvd215/github/capitalone/rubicon-ml/rubicon_ml/client/rubicon.py", line 123, in get_project
    project = self.repository.get_project(name)
  File "/Users/nvd215/github/capitalone/rubicon-ml/rubicon_ml/repository/base.py", line 104, in get_project
    raise RubiconException(f"No project with name '{project_name}' found.")
rubicon_ml.exceptions.RubiconException: No project with name 'failure modes' found.

  warnings.warn(traceback.format_exc(limit=TRACEBACK_LIMIT, chain=TRACEBACK_CHAIN))

We can also set_failure_mode to “log” to log the error with the builtin logging.error.

[3]:
set_failure_mode("log")

rb.get_project(name="failure modes")
ERROR:root:Traceback (most recent call last):
  File "/Users/nvd215/github/capitalone/rubicon-ml/rubicon_ml/client/utils/exception_handling.py", line 43, in wrapper
    return func(*args, **kwargs)
  File "/Users/nvd215/github/capitalone/rubicon-ml/rubicon_ml/client/rubicon.py", line 123, in get_project
    project = self.repository.get_project(name)
  File "/Users/nvd215/github/capitalone/rubicon-ml/rubicon_ml/repository/base.py", line 104, in get_project
    raise RubiconException(f"No project with name '{project_name}' found.")
rubicon_ml.exceptions.RubiconException: No project with name 'failure modes' found.

set_failure_mode back to the default - “raise” - to return to raising the exceptions.

Log and warning verbosity

The “log” and “warn” failure modes leverage the builtin traceback.exc_info() in order to print an error’s traceback. set_failure_mode’s traceback_limit and traceback_chain are passed directly through to the underlying call to traceback.exc_info() as the limit and chain arguments.

limit is an integer between 0 and the depth of the stack trace that controls the verbosity of the trace.

[4]:
set_failure_mode("log", traceback_limit=0)

rb.get_project(name="failure modes")
ERROR:root:rubicon_ml.exceptions.RubiconException: No project with name 'failure modes' found.

chain can be set to True to see the full chain of exceptions rather than just the final exception in the chain.

[5]:
set_failure_mode("log", traceback_chain=True)

rb.get_project(name="failure modes")
ERROR:root:Traceback (most recent call last):
  File "/Users/nvd215/mambaforge/envs/rubicon-ml-dev/lib/python3.10/site-packages/fsspec/implementations/memory.py", line 213, in cat_file
    return bytes(self.store[path].getbuffer()[start:end])
KeyError: '/root/failure-modes/metadata.json'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/nvd215/github/capitalone/rubicon-ml/rubicon_ml/repository/base.py", line 102, in get_project
    project = json.loads(self.filesystem.cat(project_metadata_path))
  File "/Users/nvd215/mambaforge/envs/rubicon-ml-dev/lib/python3.10/site-packages/fsspec/spec.py", line 755, in cat
    return self.cat_file(paths[0], **kwargs)
  File "/Users/nvd215/mambaforge/envs/rubicon-ml-dev/lib/python3.10/site-packages/fsspec/implementations/memory.py", line 215, in cat_file
    raise FileNotFoundError(path)
FileNotFoundError: /root/failure-modes/metadata.json

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/nvd215/github/capitalone/rubicon-ml/rubicon_ml/client/utils/exception_handling.py", line 43, in wrapper
    return func(*args, **kwargs)
  File "/Users/nvd215/github/capitalone/rubicon-ml/rubicon_ml/client/rubicon.py", line 123, in get_project
    project = self.repository.get_project(name)
  File "/Users/nvd215/github/capitalone/rubicon-ml/rubicon_ml/repository/base.py", line 104, in get_project
    raise RubiconException(f"No project with name '{project_name}' found.")
rubicon_ml.exceptions.RubiconException: No project with name 'failure modes' found.

Caution with return values

Some workflows require the returned rubicon-ml objects to be leveraged for future logging in the same process. For example, let’s finally create the “failure modes” project and take a look at the returned rubicon-ml object.

[6]:
rb.create_project(name="failure modes")
project = rb.get_project(name="failure modes")

print(project)
Project(name='failure modes', id='8d6ac09b-45c5-4097-9728-302c481a1665', description=None, github_url=None, training_metadata=None, created_at=datetime.datetime(2022, 11, 15, 17, 5, 2, 510729))

Now we can take any standard action on the returned rubicon-ml object, like inspecting its ID.

[7]:
print(project.id)
8d6ac09b-45c5-4097-9728-302c481a1665

If we were to leverage either the “log” or “warn” failure mode, which does not stop execution for rubicon-ml errors, we need to be cautious of returned rubicon-ml objects.

Now we’ll try to get another project that doesn’t exist. Even though this code will not stop execution after the failed get_project call, we need to be aware that a rubicon-ml project will not be returned in this case. A None will be returned in its place, thus any action taken on this returned None may fail if only rubicon-ml objects are expected downstream.

[8]:
set_failure_mode("log")

project = rb.get_project(name="failure modes v2")
print(project)
ERROR:root:Traceback (most recent call last):
  File "/Users/nvd215/github/capitalone/rubicon-ml/rubicon_ml/client/utils/exception_handling.py", line 43, in wrapper
    return func(*args, **kwargs)
  File "/Users/nvd215/github/capitalone/rubicon-ml/rubicon_ml/client/rubicon.py", line 123, in get_project
    project = self.repository.get_project(name)
  File "/Users/nvd215/github/capitalone/rubicon-ml/rubicon_ml/repository/base.py", line 104, in get_project
    raise RubiconException(f"No project with name '{project_name}' found.")
rubicon_ml.exceptions.RubiconException: No project with name 'failure modes v2' found.

None

When leveraging failure modes that do not interrupt execution, it is important to check the types of the objects returned from rubicon-ml. Simply trying to access the id attribute in this case would result in an AttributeError as a NoneType object hs no attribute id.

[9]:
if project is not None:
    print(project.id)

A more practical use case

Let’s take a look at how this may work in a more practical machine learning workflow. For this example, we’ll train a k-neighbors classifier from Scikit-learn and attempt to log the input parameters and the score the trained model produces on a test dataset.

First, we’ll create a new project and experiment to attempt to log our inputs and outputs to.

[10]:
project = rb.create_project(name="failure modes v3")
experiment = project.log_experiment()

experiment
[10]:
<rubicon_ml.client.experiment.Experiment at 0x15cacce20>

Now that we’ve got an experiment, lets replace rubicon-ml’s filesystem (the part of the library that handles actual filesystem operations) with a no-op class representing a broken filesystem. Imagine this simulating something like losing connection to S3.

We’ll also set the failure mode back to raise for this first execution of our workflow with the broken filesystem.

[11]:
class BrokenFilesystem:
    pass

rb.config.repository.filesystem = BrokenFilesystem()

set_failure_mode("raise")

Now, notice that when we run the cell below, we only fit the model before execution is halted due to experiment.log_parameter raising an exception because of the broken filesystem. We never get a score, and it is never displayed.

[12]:
from sklearn.neighbors import KNeighborsClassifier


knn = KNeighborsClassifier(n_neighbors=1)
X_train, y_train, X_test, y_test = [[0, 1, 2, 3]], [0], [[0, 1, 2, 3]], [0]

knn.fit(X_train, y_train)
experiment.log_parameter(name="n_neighbors", value=1)

score = knn.score(X_test, y_test)
experiment.log_metric(name="score", value=score)

score
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In [12], line 8
      5 X_train, y_train, X_test, y_test = [[0, 1, 2, 3]], [0], [[0, 1, 2, 3]], [0]
      7 knn.fit(X_train, y_train)
----> 8 experiment.log_parameter(name="n_neighbors", value=1)
     10 score = knn.score(X_test, y_test)
     11 experiment.log_metric(name="score", value=score)

File ~/github/capitalone/rubicon-ml/rubicon_ml/client/utils/exception_handling.py:46, in failsafe.<locals>.wrapper(*args, **kwargs)
     44 except Exception as e:
     45     if FAILURE_MODE == "raise":
---> 46         raise e
     47     elif FAILURE_MODE == "warn":
     48         warnings.warn(traceback.format_exc(limit=TRACEBACK_LIMIT, chain=TRACEBACK_CHAIN))

File ~/github/capitalone/rubicon-ml/rubicon_ml/client/utils/exception_handling.py:43, in failsafe.<locals>.wrapper(*args, **kwargs)
     41 def wrapper(*args, **kwargs):
     42     try:
---> 43         return func(*args, **kwargs)
     44     except Exception as e:
     45         if FAILURE_MODE == "raise":

File ~/github/capitalone/rubicon-ml/rubicon_ml/client/experiment.py:237, in Experiment.log_parameter(self, name, value, description, tags)
    214 """Create a parameter under the experiment.
    215
    216 Parameters
   (...)
    234     The created parameter.
    235 """
    236 parameter = domain.Parameter(name, value=value, description=description, tags=tags)
--> 237 self.repository.create_parameter(parameter, self.project.name, self.id)
    239 return Parameter(parameter, self)

File ~/github/capitalone/rubicon-ml/rubicon_ml/repository/base.py:852, in BaseRepository.create_parameter(self, parameter, project_name, experiment_id)
    836 """Persist a parameter to the configured filesystem.
    837
    838 Parameters
   (...)
    846     The ID of the experiment this parameter belongs to.
    847 """
    848 parameter_metadata_path = self._get_parameter_metadata_path(
    849     project_name, experiment_id, parameter.name
    850 )
--> 852 if self.filesystem.exists(parameter_metadata_path):
    853     raise RubiconException(f"A parameter with name '{parameter.name}' already exists.")
    855 self._persist_domain(parameter, parameter_metadata_path)

AttributeError: 'BrokenFilesystem' object has no attribute 'exists'

By setting the failure mode to “log” and ensuring we are not using any unchecked objects returned by rubicon-ml, we can ensure that the entire workflow is completed regardless of whether or not the filesystem is working.

[13]:
set_failure_mode("log")

knn = KNeighborsClassifier(n_neighbors=1)
X_train, y_train, X_test, y_test = [[0, 1, 2, 3]], [0], [[0, 1, 2, 3]], [0]

knn.fit(X_train, y_train)
experiment.log_parameter(name="n_neighbors", value=1)

score = knn.score(X_test, y_test)
experiment.log_metric(name="score", value=score)

score
ERROR:root:Traceback (most recent call last):
  File "/Users/nvd215/github/capitalone/rubicon-ml/rubicon_ml/client/utils/exception_handling.py", line 43, in wrapper
    return func(*args, **kwargs)
  File "/Users/nvd215/github/capitalone/rubicon-ml/rubicon_ml/client/experiment.py", line 237, in log_parameter
    self.repository.create_parameter(parameter, self.project.name, self.id)
  File "/Users/nvd215/github/capitalone/rubicon-ml/rubicon_ml/repository/base.py", line 852, in create_parameter
    if self.filesystem.exists(parameter_metadata_path):
AttributeError: 'BrokenFilesystem' object has no attribute 'exists'

ERROR:root:Traceback (most recent call last):
  File "/Users/nvd215/github/capitalone/rubicon-ml/rubicon_ml/client/utils/exception_handling.py", line 43, in wrapper
    return func(*args, **kwargs)
  File "/Users/nvd215/github/capitalone/rubicon-ml/rubicon_ml/client/experiment.py", line 76, in log_metric
    self.repository.create_metric(metric, self.project.name, self.id)
  File "/Users/nvd215/github/capitalone/rubicon-ml/rubicon_ml/repository/base.py", line 746, in create_metric
    if self.filesystem.exists(metric_metadata_path):
AttributeError: 'BrokenFilesystem' object has no attribute 'exists'

[13]:
1.0