Parquet Data

Contains class to save and load parquet data.

class dataprofiler.data_readers.parquet_data.ParquetData(input_file_path: Optional[str] = None, data: Optional[Union[pandas.core.frame.DataFrame, str]] = None, options: Optional[Dict] = None)

Bases: dataprofiler.data_readers.structured_mixins.SpreadSheetDataMixin, dataprofiler.data_readers.base_data.BaseData

SpreadsheetData class to save and load parquet data.

Initialize Data class for loading datasets of type PARQUET.

Can be specified by passing in memory data or via a file path. Options pertaining to PARQUET may also be specified using options dict param. Possible Options:

options = dict(
    data_format= type: str, choices: "dataframe", "records", "json"
    selected_columns= type: list(str)
    header= type: any
)

data_format: user selected format in which to return data can only be of specified types selected_columns: columns being selected from the entire dataset

Parameters

• input_file_path (str) – path to the file being loaded or None

• data (multiple types) – data being loaded into the class instead of an input file

• options (dict) – options pertaining to the data type

Returns

None

data_type: str = 'parquet'

property file_encoding: None

Set file encoding to None since not detected for avro.

property selected_columns: List[str]

Return selected columns.

property is_structured: bool

Determine compatibility with StructuredProfiler.

classmethod is_match(file_path: Union[str, _io.StringIO, _io.BytesIO], options: Optional[Dict] = None) → bool

Test the given file to check if the file has valid Parquet format.

Parameters

• file_path (str) – path to the file to be examined

• options (dict) – parquet read options

Returns

is file a parquet file or not

Return type

bool

property data

Return data.

property data_format: Optional[str]

Return data format.

get_batch_generator(batch_size: int) → Generator[Union[pandas.core.frame.DataFrame, List], None, None]

Get batch generator.

info: Optional[str] = None

property length: int

Return the length of the dataset which is loaded.

Returns

length of the dataset

reload(input_file_path: Optional[str] = None, data: Optional[Any] = None, options: Optional[Dict] = None) → None

Reload the data class with a new dataset.

This erases all existing data/options and replaces it with the input data/options.

Parameters

• input_file_path (str) – path to the file being loaded or None

• data (multiple types) – data being loaded into the class instead of an input file

• options (dict) – options pertaining to the data type

Returns

None

options: Optional[Dict]