DataComPy

DataComPy is a package to compare two DataFrames (or tables) such as Pandas, Spark, Polars, and even Snowflake. Originally it was created to be something of a replacement for SAS’s PROC COMPARE for Pandas DataFrames with some more functionality than just Pandas.DataFrame.equals(Pandas.DataFrame) (in that it prints out some stats, and lets you tweak how accurate matches have to be). Supported types include:

• Pandas

• Polars

• Spark

• Snowflake

[!IMPORTANT] datacompy has released v1. The v0.19.x line is no longer supported — users should upgrade to v1 going forward. The support/0.19.x branch is archived and will only receive critical security fixes on a best-effort basis; no new features or regular maintenance will be provided. All active development targets the v1 branches (develop and main).

Quick Installation

pip install datacompy

or

conda install datacompy

Installing extras

If you would like to use Spark or any other backends please make sure you install via extras:

pip install datacompy[spark]
pip install datacompy[snowflake]

Supported backends

• Pandas: (See documentation)

• Spark: (See documentation)

• Polars: (See documentation)

• Snowflake/Snowpark: (See documentation)

Contributors

We welcome and appreciate your contributions! Before we can accept any contributions, we ask that you please be sure to sign the Contributor License Agreement (CLA).

This project adheres to the Open Source Code of Conduct. By participating, you are expected to honor this code.

Roadmap

Roadmap details can be found here

Contents

• Installation
  • PyPI (basic)
  • conda (installs dependencies from Conda Forge)
  • virtualenv (install dependencies from PyPI)
• Comparator Framework Usage
  • Overview
  • Core Concepts
  • Creating a Custom Comparator
  • Using the Custom Comparator
• Pandas Usage
  • Overview
  • Compare Object Setup
  • Reports
  • Convenience Methods
  • Duplicate rows
  • Limitations
• Spark Usage
  • SparkSQLCompare
  • Reports
  • Convenience Methods
  • Duplicate rows
• Snowflake Usage
  • SnowflakeCompare setup
  • Reports
  • Convenience Methods
  • Duplicate rows
  • Additional considerations
• Polars Usage
  • Overview
  • Basic Usage
  • Compare Object Setup
  • Reports
  • Convenience Methods
  • Duplicate rows
  • Caveats and Limitations
• Template Guide
  • Template Basics
  • Available Template Variables
  • Creating a Custom Template
  • Example Template
  • Using a Custom Template
  • Template Path Resolution
  • Jinja2 Template Features
  • Default Template Reference
• Benchmarks
  • Setup used
  • The Data
  • TLDR
• Developer Instructions
  • Pre-Commit Hooks
  • Generating Documentation
  • Run unit tests
  • Snowflake testing
  • Management of Requirements
  • edgetest
  • Release Guide
  • Generating distribution archives (PyPI)

API Reference

• datacompy
  • datacompy package
    • Subpackages
      • datacompy.comparator package
    • Submodules
    • datacompy.base module
      • BaseCompare
      • df_to_str()
      • get_column_tolerance()
      • render()
      • save_html_report()
      • temp_column_name()
      • validate_tolerance_parameter()
    • datacompy.pandas module
      • PandasCompare
      • calculate_max_diff()
      • columns_equal()
      • generate_id_within_group()
      • get_merged_columns()
    • datacompy.polars module
      • PolarsCompare
      • calculate_max_diff()
      • columns_equal()
      • generate_id_within_group()
      • get_merged_columns()
    • datacompy.snowflake module
      • SnowflakeCompare
      • calculate_max_diff()
      • calculate_null_diff()
      • columns_equal()
      • get_merged_columns()
      • temp_column_name()
    • datacompy.spark module
      • SparkSQLCompare
      • calculate_max_diff()
      • calculate_null_diff()
      • columns_equal()
      • decimal_comparator()
      • get_merged_columns()
    • datacompy.utility module
      • check_module_available()
    • Module contents
      • BaseCompare
      • PandasCompare
      • PolarsCompare
      • SnowflakeCompare
      • SparkSQLCompare
• Github Repo

Indices and tables

• Index

• Module Index

• Search Page