Text Data

class dataprofiler.data_readers.text_data.TextData(input_file_path=None, data=None, options=None)

Bases: dataprofiler.data_readers.base_data.BaseData

TextData class to save and load text files

Data class for loading datasets of type TEXT. Can be specified by passing in memory data or via a file path. Options pertaining the TEXT may also be specified using the options dict parameter. Possible Options:

options = dict(
    data_format= type: str, choices: "text"
    samples_per_line= type: int
)

data_format: user selected format in which to return data can only be of specified types samples_per_line: chunks by which to read in the specified dataset

Parameters

• input_file_path (str) – path to the file being loaded or None

• data (multiple types) – data being loaded into the class instead of an input file

• options (dict) – options pertaining to the data type

Returns

None

data_type = 'text'

property samples_per_line

property is_structured

Determines compatibility with StructuredProfiler

tokenize()

classmethod is_match(file_path, options=None)

All files can be considered text files, hence returns True

Parameters

• file_path (str) – path to the file to be examined

• options (dict) – text file read options

Returns

is file a text file or not

Return type

bool

reload(input_file_path=None, data=None, options=None)

Reload the data class with a new dataset. This erases all existing data/options and replaces it with the input data/options.

Parameters

• input_file_path (str) – path to the file being loaded or None

• data (multiple types) – data being loaded into the class instead of an input file

• options (dict) – options pertaining to the data type

Returns

None

property data

property data_format

property file_encoding

get_batch_generator(batch_size)

info = None

property length

Returns the length of the dataset which is loaded.

Returns

length of the dataset