Base Data Labeler¶

Contains abstract classes from which labeler classes will inherit.

class dataprofiler.labelers.base_data_labeler.BaseDataLabeler(dirpath=None, load_options=None)¶

Bases: object

Parent class for data labeler objects.

Initialize DataLabeler class.

Parameters

dirpath – path to data labeler
load_options – optional arguments to include for load i.e. class for model or processors

help()¶

Describe alterable parameters.

Input data formats for preprocessors. Output data formats for postprocessors.

Returns: None

property label_mapping¶

Retrieve the label encodings.

Returns: dictionary for associating labels to indexes

property reverse_label_mapping¶

Retrieve the index to label encoding.

Returns: dictionary for associating indexes to labels

property labels¶

Retrieve the label.

Returns: list of labels

property preprocessor¶

Retrieve the data preprocessor.

Returns: returns the preprocessor instance

property model¶

Retrieve the data labeler model.

Returns: returns the model instance

property postprocessor¶

Retrieve the data postprocessor.

Returns: returns the postprocessor instance

set_params(params)¶

Allow user to set parameters of pipeline components.

Done in the following format:

params = dict(: preprocessor=dict(…), model=dict(…), postprocessor=dict(…)

)

where the key,values pairs for each pipeline component must match parameters that exist in their components.

Parameters

params (dict) –

dictionary containing a key for a given pipeline component and its associated value of parameters as such:

dict(preprocessor=dict(…), model=dict(…), postprocessor=dict(…))

Returns

None

add_label(label, same_as=None)¶

Add a label to the data labeler.

Parameters

label (str) – new label being added to the data labeler
same_as (str) – label to have the same encoding index as for multi-label to single encoding index.

Returns

None

set_labels(labels)¶

Set the labels for the data labeler.

Parameters: labels (list or dict) – new labels in either encoding list or dict
Returns: None

predict(data, batch_size=32, predict_options=None, error_on_mismatch=False, verbose=1)¶

Predict labels of input data based with the data labeler model.

Parameters

data – data to be predicted upon
batch_size – batch size of prediction
predict_options – optional parameters to allow for predict as a dict, i.e. dict(show_confidences=True)
error_on_mismatch – if true, errors instead of warns on parameter mismatches in pipeline
verbose – Flag to determine whether to print status or not

Returns

predictions

set_preprocessor(data_processor)¶

Set the data preprocessor for the data labeler.

Parameters: data_processor (data_processing.BaseDataPreprocessor) – processor to set as the preprocessor
Returns: None

set_model(model)¶

Set the model for the data labeler.

Parameters: model (base_model.BaseModel) – model to use within the data labeler
Returns: None

set_postprocessor(data_processor)¶

Set the data postprocessor for the data labeler.

Parameters: data_processor (data_processing.BaseDataPostprocessor) – processor to set as the postprocessor
Returns: None

check_pipeline(skip_postprocessor=False, error_on_mismatch=False)¶

Check whether the processors and models connect together without error.

Parameters

skip_postprocessor (bool) – skip checking postprocessor is valid in pipeline
error_on_mismatch (bool) – if true, errors instead of warns on parameter mismatches in pipeline

Returns

bool indicating valid pipeline

classmethod load_from_library(name)¶

Load the data labeler from the data labeler zoo in the library.

Parameters: name (str) – name of the data labeler.
Returns: DataLabeler class

classmethod load_from_disk(dirpath, load_options=None)¶

Load the data labeler from a saved location on disk.

Parameters

dirpath (str) – path to data labeler files.
load_options (dict) – optional arguments to include for load i.e. class for model or processors

Returns

DataLabeler class

classmethod load_with_components(preprocessor, model, postprocessor)¶

Load the data labeler from a its set of components.

Parameters

preprocessor (data_processing.BaseDataPreprocessor) – processor to set as the preprocessor
model (base_model.BaseModel) – model to use within the data labeler
postprocessor (data_processing.BaseDataPostprocessor) – processor to set as the postprocessor

Returns

save_to_disk(dirpath)¶

Save the data labeler to the specified location.

Parameters: dirpath (str) – location to save the data labeler.
Returns: None

class dataprofiler.labelers.base_data_labeler.TrainableDataLabeler(dirpath=None, load_options=None)¶

Bases: dataprofiler.labelers.base_data_labeler.BaseDataLabeler

Subclass of BaseDataLabeler that can be trained.

Initialize DataLabeler class.

Parameters

dirpath – path to data labeler
load_options – optional arguments to include for load i.e. class for model or processors

fit(x, y, validation_split=0.2, labels=None, reset_weights=False, batch_size=32, epochs=1, error_on_mismatch=False)¶

Fit the data labeler model for the dataset.

Parameters

x (Union[pd.DataFrame, pd.Series, np.ndarray]) – samples to fit model
y (Union[pd.DataFrame, pd.Series, np.ndarray]) – labels associated with the samples to fit model
validation_split (float) – split of the data to have as cross-validation data
labels (Union[list, dict]) – Encoding or number of labels if refit is needed to new labels
reset_weights (bool) – Flag to determine whether or not to reset the weights
batch_size (int) – Size of each batch sent to data labeler model
epochs (int) – number of epochs to iterate over the dataset and send to the model
error_on_mismatch (bool) – if true, errors instead of warns on parameter mismatches in pipeline

Returns

model output

set_model(model)¶

Set the model for a trainable data labeler.

Model must have a train function to be able to be set.

Parameters: model (base_model.BaseModel) – model to use within the data labeler
Returns: None

classmethod load_with_components(preprocessor, model, postprocessor)¶

Load the data labeler from a its set of components.

Parameters

preprocessor (data_processing.BaseDataPreprocessor) – processor to set as the preprocessor
model (base_model.BaseModel) – model to use within the data labeler
postprocessor (data_processing.BaseDataPostprocessor) – processor to set as the postprocessor

Returns

add_label(label, same_as=None)¶

Add a label to the data labeler.

Parameters

label (str) – new label being added to the data labeler
same_as (str) – label to have the same encoding index as for multi-label to single encoding index.

Returns

None

check_pipeline(skip_postprocessor=False, error_on_mismatch=False)¶

Check whether the processors and models connect together without error.

Parameters

skip_postprocessor (bool) – skip checking postprocessor is valid in pipeline
error_on_mismatch (bool) – if true, errors instead of warns on parameter mismatches in pipeline

Returns

bool indicating valid pipeline

help()¶

Describe alterable parameters.

Input data formats for preprocessors. Output data formats for postprocessors.

Returns: None

property label_mapping¶

Retrieve the label encodings.

Returns: dictionary for associating labels to indexes

property labels¶

Retrieve the label.

Returns: list of labels

classmethod load_from_disk(dirpath, load_options=None)¶

Load the data labeler from a saved location on disk.

Parameters

dirpath (str) – path to data labeler files.
load_options (dict) – optional arguments to include for load i.e. class for model or processors

Returns

DataLabeler class

classmethod load_from_library(name)¶

Load the data labeler from the data labeler zoo in the library.

Parameters: name (str) – name of the data labeler.
Returns: DataLabeler class

property model¶

Retrieve the data labeler model.

Returns: returns the model instance

property postprocessor¶

Retrieve the data postprocessor.

Returns: returns the postprocessor instance

predict(data, batch_size=32, predict_options=None, error_on_mismatch=False, verbose=1)¶

Predict labels of input data based with the data labeler model.

Parameters

data – data to be predicted upon
batch_size – batch size of prediction
predict_options – optional parameters to allow for predict as a dict, i.e. dict(show_confidences=True)
error_on_mismatch – if true, errors instead of warns on parameter mismatches in pipeline
verbose – Flag to determine whether to print status or not

Returns

predictions

property preprocessor¶

Retrieve the data preprocessor.

Returns: returns the preprocessor instance

property reverse_label_mapping¶

Retrieve the index to label encoding.

Returns: dictionary for associating indexes to labels

save_to_disk(dirpath)¶

Save the data labeler to the specified location.

Parameters: dirpath (str) – location to save the data labeler.
Returns: None

set_labels(labels)¶

Set the labels for the data labeler.

Parameters: labels (list or dict) – new labels in either encoding list or dict
Returns: None

set_params(params)¶

Allow user to set parameters of pipeline components.

Done in the following format:

params = dict(: preprocessor=dict(…), model=dict(…), postprocessor=dict(…)

)

where the key,values pairs for each pipeline component must match parameters that exist in their components.

Parameters

params (dict) –

dictionary containing a key for a given pipeline component and its associated value of parameters as such:

dict(preprocessor=dict(…), model=dict(…), postprocessor=dict(…))

Returns

None

set_postprocessor(data_processor)¶

Set the data postprocessor for the data labeler.

Parameters: data_processor (data_processing.BaseDataPostprocessor) – processor to set as the postprocessor
Returns: None

set_preprocessor(data_processor)¶

Set the data preprocessor for the data labeler.

Parameters: data_processor (data_processing.BaseDataPreprocessor) – processor to set as the preprocessor
Returns: None