View this notebook on GitHub

Graph Pipeline Demo

DataProfiler can also load and profile graph datasets. Similarly to the rest of DataProfiler profilers, this is split into two components:

• GraphData

• GraphProfiler

We will demo the use of this graph pipeline.

First, let’s import the libraries needed for this example.

import os
import sys
import pprint

try:
    sys.path.insert(0, '..')
    import dataprofiler as dp
except ImportError:
    import dataprofiler as dp

data_path = "../dataprofiler/tests/data"

We now input our dataset into the generic DataProfiler pipeline:

data = dp.Data(os.path.join(data_path, "csv/graph_data_csv_identify.csv"))
profile = dp.Profiler(data)

report = profile.report()

pp = pprint.PrettyPrinter(sort_dicts=False, compact=True)
pp.pprint(report)

We notice that the Data class automatically detected the input file as graph data. The GraphData class is able to differentiate between tabular and graph csv data. After Data matches the input file as graph data, GraphData does the necessary work to load the csv data into a NetworkX Graph.

Profiler runs GraphProfiler when graph data is input (or when data_type="graph" is specified). The report() function outputs the profile for the user.

Profile

The profile skeleton looks like this:

profile = {
    "num_nodes": ...,
    "num_edges": ...,
    "categorical_attributes": ...,
    "continuous_attributes": ...,
    "avg_node_degree": ...,
    "global_max_component_size": ...,
    "continuous_distribution": ...,
    "categorical_distribution": ...,
    "times": ...,
}

Description of properties in profile:

• num_nodes: number of nodes in the graph

• num_edges: number of edges in the graph

• categorical_attributes: list of categorical edge attributes

• continuous_attributes: list of continuous edge attributes

• avg_node_degree: average degree of nodes in the graph

• global_max_component_size: size of largest global max component in the graph

• continuous_distribution: dictionary of statistical properties for each continuous attribute

• categorical_distribution: dictionary of statistical properties for each categorical attribute

The continuous_distribution and categorical_distribution dictionaries list statistical properties for each edge attribute in the graph:

continuous_distribution = {
    "name": ...,
    "scale": ...,
    "properties": ...,
}

categorical_distribution = {
    "bin_counts": ...,
    "bin_edges": ...,
}

Description of each attribute:

• Continuous distribution:

  • name: name of the distribution

  • scale: negative log likelihood used to scale distributions and compare them in GraphProfiler

  • properties: list of distribution props

• Categorical distribution:

  • bin_counts: histogram bin counts

  • bin_edges: histogram bin edges

properties lists the following distribution properties: [optional: shape, loc, scale, mean, variance, skew, kurtosis]. The list can be either 6 length or 7 length depending on the distribution (extra shape parameter):

• 6 length: norm, uniform, expon, logistic

• 7 length: gamma, lognorm

  • gamma: shape=a (float)

  • lognorm: shape=s (float)

For more information on shape parameters a and s: https://docs.scipy.org/doc/scipy/tutorial/stats.html#shape-parameters

Saving and Loading a Profile

Below you will see an example of how a Graph Profile can be saved and loaded again.

# The default save filepath is profile-<datetime>.pkl
profile.save(filepath="profile.pkl")

new_profile = dp.GraphProfiler.load("profile.pkl")
new_report = new_profile.report()

pp.pprint(report)

Difference in Data

If we wanted to ensure that this new profile was the same as the previous profile that we loaded, we could compare them using the diff functionality.

diff = profile.diff(new_profile)

pp.pprint(diff)

Another use for diff might be to provide differences between training and testing profiles as shown in the cell below. We will use the profile above as the training profile and create a new profile to represent the testing profile

training_profile = profile

testing_data = dp.Data(os.path.join(data_path, "csv/graph-differentiator-input-positive.csv"))
testing_profile = dp.Profiler(testing_data)

test_train_diff = training_profile.diff(testing_profile)

Below you can observe the difference between the two profiles.

pp.pprint(test_train_diff)

Conclusion

We have shown the graph pipeline in the DataProfiler. It works similarly to the current DataProfiler implementation.