Skip to main content
Version: v0.21

DBT

Summary​

dbt (data build tool) enables analytics engineers to transform data in their warehouses by simply writing select statements. dbt handles turning these select statements into tables and views. dbt does the T in ELT (Extract, Load, Transform) processes – it doesn’t extract or load data, but it’s extremely good at transforming data that’s already loaded into your warehouse.

User setup​

  • If you plan to use this product, you need to install some environments first.

Required Packages to Install​

Commands to run or create in your terminal and the dbt project​

  1. pip install dbt-mysql
  2. dbt init demoapp (demoapp is project name)
  3. create your SQL transformations and data models

Convert Data By DBT​

Use the Raw JSON API to manually initiate a run using cURL or graphical API tool such as Postman. POST the following request to the DevLake API Endpoint.

[
  [
    {
      "plugin": "dbt",
      "options": {
          "projectPath": "/Users/abeizn/demoapp",
          "projectName": "demoapp",
          "projectTarget": "dev",
          "selectedModels": ["my_first_dbt_model","my_second_dbt_model"],
          "projectVars": {
            "demokey1": "demovalue1",
            "demokey2": "demovalue2"
        }
      }
    }
  ]
]
  • projectPath: the absolute path of the dbt project. (required)
  • projectName: the name of the dbt project. (required)
  • projectTarget: this is the default target your dbt project will use. (optional)
  • selectedModels: a model is a select statement. Models are defined in .sql files, and typically in your models directory. (required) And selectedModels accepts one or more arguments. Each argument can be one of:
  1. a package name, runs all models in your project, example: example
  2. a model name, runs a specific model, example: my_fisrt_dbt_model
  3. a fully-qualified path to a directory of models.
  • projectVars: variables to parametrize dbt models. (optional) example: select * from events where event_type = '{{ var("event_type") }}' To execute this SQL query in your model, you need set a value for event_type.

Resources:​