Text Data¶
Contains class for saving and loading text files.
- class dataprofiler.data_readers.text_data.TextData(input_file_path=None, data=None, options=None)¶
Bases:
dataprofiler.data_readers.base_data.BaseData
TextData class to save and load text files.
Initialize Data class for loading datasets of type TEXT.
Can be specified by passing in memory data or via a file path. Options pertaining the TEXT may also be specified using the options dict parameter. Possible Options:
options = dict( data_format= type: str, choices: "text" samples_per_line= type: int )
data_format: user selected format in which to return data can only be of specified types samples_per_line: chunks by which to read in the specified dataset
- Parameters
input_file_path (str) – path to the file being loaded or None
data (multiple types) – data being loaded into the class instead of an input file
options (dict) – options pertaining to the data type
- Returns
None
- data_type = 'text'¶
- property samples_per_line¶
Return samples per line.
- property is_structured¶
Determine compatibility with StructuredProfiler.
- tokenize()¶
Tokenize data.
- classmethod is_match(file_path, options=None)¶
Return True if all are text files.
- Parameters
file_path (str) – path to the file to be examined
options (dict) – text file read options
- Returns
is file a text file or not
- Return type
bool
- reload(input_file_path=None, data=None, options=None)¶
Reload the data class with a new dataset.
This erases all existing data/options and replaces it with the input data/options.
- Parameters
input_file_path (str) – path to the file being loaded or None
data (multiple types) – data being loaded into the class instead of an input file
options (dict) – options pertaining to the data type
- Returns
None
- property data¶
Return data.
- property data_format¶
Return data format.
- property file_encoding¶
Return file encoding.
- get_batch_generator(batch_size)¶
Get batch generator.
- info = None¶
- property length¶
Return the length of the dataset which is loaded.
- Returns
length of the dataset