TobikoData/sqlmesh

SQLMesh is a next-generation data transformation framework designed to ship data quickly, efficiently, and without error. Data teams can efficiently run and deploy data transformations written in SQL or Python with visibility and control at any size.

It is more than just a dbt alternative.

Core Features

Get instant SQL impact analysis of your changes, whether in the CLI or in SQLMesh Plan Mode

• Create isolated development environments without data warehouse costs
• Plan / Apply workflow like Terraform to understand potential impact of changes
• Easy to use CI/CD bot for true blue-green deployments

sqlmesh create_test tcloud_demo.stg_payments --query tcloud_demo.seed_raw_payments "select * from tcloud_demo.seed_raw_payments limit 5"

# run the unit test
sqlmesh test

MODEL (
  name tcloud_demo.stg_payments,
  cron '@daily',
  grain payment_id,
  audits (UNIQUE_VALUES(columns = (
      payment_id
  )), NOT_NULL(columns = (
      payment_id
  )))
);

SELECT
    id AS payment_id,
    order_id,
    payment_method,
    amount / 100 AS amount, /* `amount` is currently stored in cents, so we convert it to dollars */
    'new_column' AS new_column, /* non-breaking change example  */
FROM tcloud_demo.seed_raw_payments

test_stg_payments:
model: tcloud_demo.stg_payments
inputs:
    tcloud_demo.seed_raw_payments:
      - id: 66
        order_id: 58
        payment_method: coupon
        amount: 1800
      - id: 27
        order_id: 24
        payment_method: coupon
        amount: 2600
      - id: 30
        order_id: 25
        payment_method: coupon
        amount: 1600
      - id: 109
        order_id: 95
        payment_method: coupon
        amount: 2400
      - id: 3
        order_id: 3
        payment_method: coupon
        amount: 100
outputs:
    query:
      - payment_id: 66
        order_id: 58
        payment_method: coupon
        amount: 18.0
        new_column: new_column
      - payment_id: 27
        order_id: 24
        payment_method: coupon
        amount: 26.0
        new_column: new_column
      - payment_id: 30
        order_id: 25
        payment_method: coupon
        amount: 16.0
        new_column: new_column
      - payment_id: 109
        order_id: 95
        payment_method: coupon
        amount: 24.0
        new_column: new_column
      - payment_id: 3
        order_id: 3
        payment_method: coupon
        amount: 1.0
        new_column: new_column

• Never build a table more than once
• Track what data’s been modified and run only the necessary transformations for incremental models
• Run unit tests for free and configure automated audits

Level Up Your SQL

Write SQL in any dialect and SQLMesh will transpile it to your target SQL dialect on the fly before sending it to the warehouse.

• Debug transformation errors before you run them in your warehouse in 10+ different SQL dialects
• Definitions using simply SQL (no need for redundant and confusing Jinja + YAML)
• See impact of changes before you run them in your warehouse with column-level lineage

For more information, check out the website and documentation.

Getting Started

Install SQLMesh through pypi by running:

mkdir sqlmesh-example
cd sqlmesh-example
python -m venv .venv
source .venv/bin/activate
pip install sqlmesh
source .venv/bin/activate # reactivate the venv to ensure you're using the right installation
sqlmesh init duckdb # get started right away with a local duckdb instance
sqlmesh plan # see the plan for the changes you're making

Note: You may need to run python3 or pip3 instead of python or pip, depending on your python installation.

Follow the quickstart guide to learn how to use SQLMesh. You already have a head start!

Follow this example to learn how to use SQLMesh in a full walkthrough.

Join Our Community

Together, we want to build data transformation without the waste. Connect with us in the following ways:

• Join the Tobiko Slack Community to ask questions, or just to say hi!
• File an issue on our GitHub
• Send us an email at hello@tobikodata.com with your questions or feedback
• Read our blog

Contribution

Contributions in the form of issues or pull requests (from fork) are greatly appreciated.

Read more on how to contribute to SQLMesh open source.

Watch this video walkthrough to see how our team contributes a feature to SQLMesh.