DataComPy

DataComPy is a package to compare two Pandas DataFrames. Originally started to be something of a replacement for SAS’s PROC COMPARE for Pandas DataFrames with some more functionality than just Pandas.DataFrame.equals(Pandas.DataFrame) (in that it prints out some stats, and lets you tweak how accurate matches have to be). Then extended to carry that functionality over to Spark Dataframes.

Quick Installation

pip install datacompy

or

conda install datacompy

Installing extras

If you would like to use Spark or any other backends please make sure you install via extras:

pip install datacompy[spark]
pip install datacompy[fugue]
pip install datacompy[snowflake]

Legacy Spark Deprecation

With version v0.12.0 the original SparkCompare was replaced with a Pandas on Spark implementation. The original SparkCompare implementation differs from all the other native implementations. To align the API better, and keep behaviour consistent we are deprecating the original SparkCompare into a new module LegacySparkCompare

Subsequently in v0.13.0 a PySpark DataFrame class has been introduced (SparkSQLCompare) which accepts pyspark.sql.DataFrame and should provide better performance. With this version the Pandas on Spark implementation has been renamed to SparkPandasCompare and all the spark logic is now under the spark submodule.

If you wish to use the old SparkCompare moving forward you can import it like so:

from datacompy.spark.legacy import LegacySparkCompare

SparkPandasCompare Deprecation

Starting with v0.14.1, SparkPandasCompare is slated for deprecation. SparkSQLCompare is the prefered and much more performant. It should be noted that if you continue to use SparkPandasCompare that numpy 2+ is not supported due to dependency issues.

Supported versions and dependncies

Different versions of Spark, Pandas, and Python interact differently. Below is a matrix of what we test with. With the move to Pandas on Spark API and compatability issues with Pandas 2+ we will for the mean time note support Pandas 2 with the Pandas on Spark implementation. Spark plans to support Pandas 2 in Spark 4

	Spark 3.2.4	Spark 3.3.4	Spark 3.4.2	Spark 3.5.1
Python 3.9	✅	✅	✅	✅
Python 3.10	✅	✅	✅	✅
Python 3.11	❌	❌	✅	✅
Python 3.12	❌	❌	❌	❌

	Pandas < 1.5.3	Pandas >=2.0.0
Compare	✅	✅
SparkPandasCompare	✅	❌
SparkSQLCompare	✅	✅
Fugue	✅	✅

[!NOTE] At the current time Python 3.12 is not supported by Spark and also Ray within Fugue. If you are using Python 3.12 and above, please note that not all functioanlity will be supported. Pandas and Polars support should work fine and are tested.

Supported backends

• Pandas: (See documentation)

• Spark: (See documentation)

• Polars: (See documentation)

• Snowflake/Snowpark: (See documentation)

• Fugue is a Python library that provides a unified interface for data processing on Pandas, DuckDB, Polars, Arrow, Spark, Dask, Ray, and many other backends. DataComPy integrates with Fugue to provide a simple way to compare data across these backends. Please note that Fugue will use the Pandas (Native) logic at its lowest level (See documentation)

Contributors

We welcome and appreciate your contributions! Before we can accept any contributions, we ask that you please be sure to sign the Contributor License Agreement (CLA).

This project adheres to the Open Source Code of Conduct. By participating, you are expected to honor this code.

Roadmap

Roadmap details can be found here

Contents

• Installation
  • PyPI (basic)
  • conda (installs dependencies from Conda Forge)
  • virtualenv (install dependencies from PyPI)
• Pandas Usage
  • Overview
  • Compare Object Setup
  • Reports
  • Convenience Methods
  • Duplicate rows
  • Limitations
• Spark Usage
  • SparkSQLCompare
  • Reports
  • Convenience Methods
  • Duplicate rows
• Snowflake Usage
  • SnowflakeCompare setup
  • Reports
  • Convenience Methods
  • Duplicate rows
  • Additional considerations
• Polars Usage
  • Overview
  • Basic Usage
  • Compare Object Setup
  • Reports
  • Convenience Methods
  • Duplicate rows
  • Caveats and Limitations
• Fugue Usage
  • Installation
  • Basic Usage
  • Cross Comparing
  • How it works
• Benchmarks
  • Setup used
  • The Data
  • TLDR
• Developer Instructions
  • Pre-Commit Hooks
  • Generating Documentation
  • Run unit tests
  • Snowflake testing
  • Management of Requirements
  • edgetest
  • Release Guide
  • Generating distribution archives (PyPI)

API Reference

• datacompy
  • datacompy package
    • Subpackages
      • datacompy.spark package
    • Submodules
    • datacompy.base module
      • BaseCompare
      • temp_column_name()
    • datacompy.core module
      • Compare
      • calculate_max_diff()
      • columns_equal()
      • compare_string_and_date_columns()
      • generate_id_within_group()
      • get_merged_columns()
      • render()
    • datacompy.fugue module
      • all_columns_match()
      • all_rows_overlap()
      • count_matching_rows()
      • intersect_columns()
      • is_match()
      • report()
      • unq_columns()
    • datacompy.logger module
      • get_logger()
    • datacompy.polars module
      • PolarsCompare
      • calculate_max_diff()
      • columns_equal()
      • compare_string_and_date_columns()
      • generate_id_within_group()
      • get_merged_columns()
      • render()
    • datacompy.snowflake module
      • SnowflakeCompare
      • calculate_max_diff()
      • calculate_null_diff()
      • columns_equal()
      • get_merged_columns()
      • render()
      • temp_column_name()
    • Module contents
      • BaseCompare
      • Compare
      • PolarsCompare
      • SnowflakeCompare
      • SparkPandasCompare
      • SparkSQLCompare
      • all_columns_match()
      • all_rows_overlap()
      • calculate_max_diff()
      • columns_equal()
      • compare_string_and_date_columns()
      • count_matching_rows()
      • generate_id_within_group()
      • get_merged_columns()
      • intersect_columns()
      • is_match()
      • render()
      • report()
      • temp_column_name()
      • unq_columns()
• Github Repo

Indices and tables

• Index

• Module Index

• Search Page