Basic knowledge of AWS ECS, Terraform, and your network where your cluster will be deployed is necessary to follow this guide effectively.

Setting up terraform structure

To configure devlake, we will use a terraform module and pass the necessary variables. The final structure of the module would be something like this:

terraform-root/
├── main.tf
├── ...
├── modules/
│   ├── devlake/
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   ├── outputs.tf
│   │   ├── rds.tf
│   │   ├── alb.tf
│   │   ├── efs.tf
│   │   ├── iam.tf
│   │   ├── logs.tf
│   │   ├── security.tf
│   │   └── providers.tf
│   │

As we go in this tutorial, we will be populating individual files.

The AWS provider that is being used is version = "~> 5.91.0".

Fist of, lets populate the variables.tf:

variable "aws_region" {
  description = "AWS region where resources will be created"
  type        = string
  default     = "YOUR-AWS-REGION" 
}

variable "tag_name" { # for cost monitoring
  description = "Name tag for all DevLake resources"
  type        = string
  default     = "devlake"
}

High level architecture

The goal is to create an AWS ECS cluster running on fargate spot instances, with an RDS mysql database and efs for grafana as a storage layer. We also need an application load balancer on top which we can optionally configure authentication (like OKTA).

Storage​

Let's start with provisioning the necessary storage layer. First of, let's create a database and tis security group in the rds.tf file. For that we will need the subnets that you want to deploy your database, the vpc id and the database password (adding them to the variable.tf):

module "devlake-rds" {
  source  = "terraform-aws-modules/rds/aws"
  version = "~> 6.10.0"

  identifier = "devlake-db"

  engine               = "mysql"
  engine_version       = "8.0"
  instance_class       = "db.t3.micro"
  allocated_storage    = 20
  major_engine_version = "8.0"
  family               = "mysql8.0"

  manage_master_user_password = false 
  db_name                     = "devlake"
  username                    = "devlake"
  password                    = var.db_password # todo
  port                        = 3306

  create_db_subnet_group = true
  subnet_ids             = var.private_subnets # todo
  vpc_security_group_ids = [aws_security_group.rds.id]

  skip_final_snapshot = true

  enabled_cloudwatch_logs_exports = ["error", "general", "slowquery"]
  create_cloudwatch_log_group     = true

  tags = {
    Name = var.tag_name
  }
}

resource "aws_security_group" "rds" {
  name        = "devlake-rds"
  description = "Security group for DevLake RDS"
  vpc_id      = var.vpc_id # todo

  tags = {
    Name = var.tag_name
  }

  ingress {
    from_port       = 3306
    to_port         = 3306
    protocol        = "tcp"
    security_groups = [aws_security_group.ecs_tasks.id]
  }
}

And also let's create the EFS for grafana and its own security group in efs.tf. Since we know that grafana will also need to access this efs, lets go ahead and also create the access point and mount targets:

resource "aws_efs_file_system" "grafana" {
  creation_token = "grafana-storage"
  encrypted      = true

  lifecycle_policy {
    transition_to_ia = "AFTER_30_DAYS"
  }

  tags = {
    Name = var.tag_name
  }
}

resource "aws_efs_access_point" "grafana" {
  file_system_id = aws_efs_file_system.grafana.id

  posix_user {
    gid = 472
    uid = 472
  }

  root_directory {
    path = "/var/lib/grafana" # from devlake documentation
    creation_info { # this is necessary to give write permissions
      owner_gid   = 472
      owner_uid   = 472
      permissions = "755"
    }
  }

  tags = {
    Name = var.tag_name
  }
}

resource "aws_efs_mount_target" "grafana" {
  count           = length(var.private_subnets)
  file_system_id  = aws_efs_file_system.grafana.id
  subnet_id       = var.private_subnets[count.index]
  security_groups = [aws_security_group.efs.id]
}

resource "aws_security_group" "efs" {
  name        = "grafana-efs"
  description = "Security group for Grafana EFS mount targets"
  vpc_id      = var.vpc_id

  tags = {
    Name = var.tag_name
  }

  ingress {
    from_port       = 2049
    to_port         = 2049
    protocol        = "tcp"
    security_groups = [aws_security_group.ecs_tasks.id]
  }
} 

Cluster​

Let's start with creating a cluster in the module's main.tf:

resource "aws_ecs_cluster" "devlake" {
  name = "devlake-cluster"

  setting {
    name  = "containerInsights"
    value = "enabled"
  }

  tags = {
    Name = var.tag_name
  }
}

We also need to make sure that the services can talk to each other, therefore we need to create a service discovery namespace. This is crucial to allow for proper host resolution (side note, I also tested this with service connect and it did not work):

resource "aws_service_discovery_private_dns_namespace" "devlake" {
  name        = "devlake-ns"
  description = "Private DNS namespace for DevLake services"
  vpc         = var.vpc_id
  tags = {
    Name = var.tag_name
  }
}

# one for each service
resource "aws_service_discovery_service" "devlake" {
  name = "devlake" // "config-ui" // "grafana"

  dns_config {
    namespace_id = aws_service_discovery_private_dns_namespace.devlake.id

    dns_records {
      ttl  = 10
      type = "A"
    }

    routing_policy = "MULTIVALUE"
  }

  health_check_custom_config {
    failure_threshold = 1
  }

  tags = {
    Name = var.tag_name
  }
}

Permissions​

We also need to create IAM roles with necessary permissions to run our cluster. And a separate task role for the grafana container for efs permissions:

resource "aws_iam_role" "ecs_task_execution_role" {
  name = "devlake-ecs-task-execution-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "ecs-tasks.amazonaws.com"
        }
      }
    ]
  })

  tags = {
    Name = var.tag_name
  }
}

resource "aws_iam_role_policy_attachment" "ecs_task_execution_role_policy" {
  role       = aws_iam_role.ecs_task_execution_role.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
}

resource "aws_iam_role_policy_attachment" "ecs_task_execution_role_policy_cloudwatch" {
  role       = aws_iam_role.ecs_task_execution_role.name
  policy_arn = "arn:aws:iam::aws:policy/CloudWatchLogsFullAccess"
}

# IAM Task Role for Grafana container
resource "aws_iam_role" "grafana_task_role" {
  name = "devlake-grafana-task-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "ecs-tasks.amazonaws.com"
        }
      }
    ]
  })

  tags = {
    Name = var.tag_name
  }
}

# Policy allowing Grafana task to access EFS
resource "aws_iam_role_policy" "grafana_efs_access_policy" {
  name = "grafana-efs-access-policy"
  role = aws_iam_role.grafana_task_role.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "elasticfilesystem:ClientMount",
          "elasticfilesystem:ClientWrite",
          "elasticfilesystem:DescribeAccessPoints",
          "elasticfilesystem:DescribeFileSystems"
        ]
        Resource = "*"
      }
    ]
  })
}

Task definition​

Since the task definition is more or less identical, I will provide an example for one and just the differences in configuration for others.

resource "aws_ecs_task_definition" "devlake" {
  family                   = "devlake"
  network_mode             = "awsvpc"
  requires_compatibilities = ["FARGATE"]
  cpu                      = 256
  memory                   = 512
  execution_role_arn       = aws_iam_role.ecs_task_execution_role.arn

  container_definitions = jsonencode([
    {
      name  = "devlake"
      image = "devlake.docker.scarf.sh/apache/devlake:v1.0.1"
      portMappings = [
        {
          name          = "devlake-port" 
          containerPort = 8080
          hostPort      = 8080
          protocol      = "tcp"
        }
      ]
      environment = [
        {
          name  = "DB_URL"
          value = "mysql://devlake:${var.db_password}@${module.devlake-rds.db_instance_endpoint}/devlake?charset=utf8mb4&parseTime=True&loc=UTC"
        },
        {
          name  = "LOGGING_DIR"
          value = "/app/logs"
        },
        {
          name  = "TZ"
          value = "UTC"
        },
        {
          name  = "ENCRYPTION_SECRET"
          value = var.encryption_secret # your encryption secret
        },
      ]
      logConfiguration = {
        logDriver = "awslogs"
        options = {
          "awslogs-group"         = aws_cloudwatch_log_group.devlake.name
          "awslogs-region"        = var.aws_region
          "awslogs-stream-prefix" = "devlake"
        }
      }
    }
  ])

  tags = {
    Name = var.tag_name
  }
}

for config-ui, all is same except for the definition. Take a note on how the endpoints look like, as they are using the namespace we created before:

resource "aws_ecs_task_definition" "config_ui" {
  family                   = "config-ui"
  ...
  container_definitions = jsonencode([
    {
      name  = "config-ui"
      image = "devlake.docker.scarf.sh/apache/devlake-config-ui:latest"
      portMappings = [
        {
          name          = "config-ui-port" 
          containerPort = 4000
          hostPort      = 4000
          protocol      = "tcp"
        }
      ]
      environment = [
        {
          "name" : "DEVLAKE_ENDPOINT",
          "value" : "devlake.devlake-ns:8080"
        },
        {
          name  = "GRAFANA_ENDPOINT"
          value = "grafana.devlake-ns:3000"
        },
        {
          name  = "TZ"
          value = "UTC"
        }
      ]
      logConfiguration = {
        logDriver = "awslogs"
        options = {
          "awslogs-group"         = aws_cloudwatch_log_group.config_ui.name
          "awslogs-region"        = var.aws_region
          "awslogs-stream-prefix" = "config-ui"
        }
      }
    }
  ])
  ...
}

and for grafana, we need to define the task role as well as the volume mount:

resource "aws_ecs_task_definition" "grafana" {
  family                   = "grafana"
  ...
  task_role_arn            = aws_iam_role.grafana_task_role.arn

  container_definitions = jsonencode([
    {
      name  = "grafana"
      image = "devlake.docker.scarf.sh/apache/devlake-dashboard:v1.0.1"
      portMappings = [
        {
          name          = "grafana-port" 
          containerPort = 3000
          hostPort      = 3000
          protocol      = "tcp"
        }
      ]
      environment = [
        {
          name  = "GF_SERVER_ROOT_URL"
          value = "https://devlake.${var.domain_name}/grafana"
        },
        {
          name  = "TZ"
          value = "UTC"
        },
        {
          name  = "MYSQL_URL"
          value = module.devlake-rds.db_instance_endpoint
        },
        {
          name  = "MYSQL_DATABASE"
          value = "devlake"
        },
        {
          name  = "MYSQL_USER"
          value = "devlake"
        },
        {
          name  = "MYSQL_PASSWORD"
          value = var.db_password
        }
      ]
      mountPoints = [
        {
          sourceVolume  = "grafana-storage"
          containerPath = "/var/lib/grafana"
          readOnly      = false
        }
      ]
      logConfiguration = {
        logDriver = "awslogs"
        options = {
          "awslogs-group"         = aws_cloudwatch_log_group.grafana.name
          "awslogs-region"        = var.aws_region
          "awslogs-stream-prefix" = "grafana"
        }
      }
    }
  ])

  volume {
    name = "grafana-storage"
    efs_volume_configuration {
      file_system_id     = aws_efs_file_system.grafana.id
      root_directory     = "/"
      transit_encryption = "ENABLED"
      authorization_config {
        access_point_id = aws_efs_access_point.grafana.id
        iam             = "ENABLED"
      }
    }
  }
  ...
}

Security​

To create the service we also need to create a task security group in security.tf.

resource "aws_security_group" "ecs_tasks" {
  name        = "devlake-ecs-tasks"
  description = "Security group for ECS tasks"
  vpc_id      = var.management_vpc.vpc_id

  ingress {
    from_port   = 8080
    to_port     = 8080
    protocol    = "tcp"
    self        = true
    description = "Allow traffic from other ECS tasks to DevLake API"
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name = var.tag_name
  }
}

We also need to add ingress rules to the RDS and EFS security groups in rds.tf and efs.tf so that the tasks can reach these services.

resource "aws_security_group" "rds" {
  ...
  ingress {
    from_port       = 3306
    to_port         = 3306
    protocol        = "tcp"
    security_groups = [aws_security_group.ecs_tasks.id]
  }
  ...
}

resource "aws_security_group" "efs" {
  ...
  ingress {
    from_port       = 2049
    to_port         = 2049
    protocol        = "tcp"
    security_groups = [aws_security_group.ecs_tasks.id]
  }
  ...
} 

Logs​

To ensure proper monitoring and debugging capabilities, we'll set up CloudWatch log groups for each service.

resource "aws_cloudwatch_log_group" "devlake" {
  name              = "/ecs/devlake"
  retention_in_days = 30

  tags = {
    Name = var.tag_name
  }
}

resource "aws_cloudwatch_log_group" "config_ui" {
  name              = "/ecs/config-ui"
  retention_in_days = 30

  tags = {
    Name = var.tag_name
  }
}

resource "aws_cloudwatch_log_group" "grafana" {
  name              = "/ecs/grafana"
  retention_in_days = 30

  tags = {
    Name = var.tag_name
  }
}

Services​

Now that we have our infrastructure and task definitions set up, let's create the ECS services. We'll deploy three services: DevLake, Config UI, and Grafana. Each service will use Fargate Spot instances for cost optimization and will be configured with service discovery for internal communication.

resource "aws_ecs_service" "devlake" {
  name            = "devlake"
  cluster         = aws_ecs_cluster.devlake.id
  task_definition = aws_ecs_task_definition.devlake.arn
  desired_count   = 1

  health_check_grace_period_seconds = 120 # 2 minutes grace period for startup

  capacity_provider_strategy {
    base              = 1
    weight            = 100
    capacity_provider = "FARGATE_SPOT"
  }

  network_configuration {
    subnets         = var.management_vpc.private_subnets
    security_groups = [aws_security_group.ecs_tasks.id]
  }

  # Configure Service Discovery
  service_registries {
    registry_arn = aws_service_discovery_service.devlake.arn
  }

  deployment_circuit_breaker {
    enable   = true
    rollback = true
  }

  # Configure deployment settings
  deployment_maximum_percent         = 100
  deployment_minimum_healthy_percent = 0

  propagate_tags = "SERVICE"
  tags = {
    Name = var.tag_name
  }
}

Take a note of the deployment setting in devlake service configuration. This is not needed for config ui or grafana, but only for devlake as only one instance od devlake can connect to DB, therefore we need to first kill the instance completely before starting a new one.

resource "aws_ecs_service" "config_ui" {
  name            = "config-ui"
  task_definition = aws_ecs_task_definition.config_ui.arn
  
  ...

  service_registries {
    registry_arn = aws_service_discovery_service.config_ui.arn
  }

  # Wait for DevLake to be ready
  depends_on = [aws_ecs_service.devlake]
}

resource "aws_ecs_service" "grafana" {
  name            = "grafana"
  task_definition = aws_ecs_task_definition.grafana.arn
  
  ...

  service_registries {
    registry_arn = aws_service_discovery_service.grafana.arn
  }

  # Wait for both DevLake and Config UI to be ready
  depends_on = [aws_ecs_service.devlake, aws_ecs_service.config_ui]
}

At this point your containers should be up and running, without any errors regarding DB connection or write permissions to efs. Now we need to setup the load balancer and domain name.

Load balancer​

Load balancer helps us route secure traffic to correct containers. We want to have a human readable url, like https://devlake.YOUR-DOMAIN.com to access config ui and https://devlake.YOUR-DOMAIN.com/grafana to access grafana dashboards.

Security​

Firts we need to create a security group for our alb in security.tf:

resource "aws_security_group" "alb" {
  name        = "devlake-alb"
  description = "Security group for DevLake ALB"
  vpc_id      = var.vpc_id

  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
    description = "Allow HTTP traffic"
  }

  ingress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
    description = "Allow HTTPS traffic"
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
    description = "Allow all outbound traffic"
  }

  tags = {
    Name = var.tag_name
  }
} 

We also need to allow traffic from alb to the task security groups. This creates a secure path for external traffic to reach our services:

resource "aws_security_group" "ecs_tasks" {
  ...

  ingress {
    from_port       = 8080
    to_port         = 8080
    protocol        = "tcp"
    security_groups = [aws_security_group.alb.id]
    description     = "Allow traffic from ALB to DevLake API"
  }

  ingress {
    from_port       = 4000
    to_port         = 4000
    protocol        = "tcp"
    security_groups = [aws_security_group.alb.id]
    description     = "Allow traffic from ALB to DevLake UI"
  }

  ingress {
    from_port       = 3000
    to_port         = 3000
    protocol        = "tcp"
    security_groups = [aws_security_group.alb.id]
    description     = "Allow traffic from ALB to Grafana UI"
  }
  ...
}

Certificate​

To create an alb we also need a valid certificate. The certificate must be in the same region as the alb. This are probably created outside of the devlake module.

resource "aws_acm_certificate" "devlake_cert" {
  domain_name       = "*.YOUR-DOMAIN.COM"
  validation_method = "DNS"

  lifecycle {
    create_before_destroy = true
  }

  tags = {
    Name = "devlake"
  }
}

resource "aws_route53_record" "cert_validation" {
  for_each = {
    for dvo in aws_acm_certificate.devlake_cert.domain_validation_options : dvo.domain_name => {
      name   = dvo.resource_record_name
      record = dvo.resource_record_value
      type   = dvo.resource_record_type
    }
  }

  allow_overwrite = true
  zone_id         = YOUR_DOMAIN_ROUTE53.zone_id
  name            = each.value.name
  type            = each.value.type
  records         = [each.value.record]
  ttl             = 60
}

ALB​

To create alb, we need two variables - public subnets and certificate created in the previous step.

Target groups are essential components that route traffic to our ECS tasks. We'll create two target groups:

1. Grafana target group - routes traffic to port 3000 and checks health at /api/health
2. Config UI target group - routes traffic to port 4000 and checks health at the root path /

The health checks ensure that traffic is only routed to healthy containers, maintaining service reliability.

Listeners define how the ALB routes incoming traffic to our target groups.

The HTTPS listener uses our ACM certificate and routes traffic based on host headers:

• devlake.${var.domain_name}/grafana* → Grafana target group
• devlake.${var.domain_name} → Config UI target group

This setup ensures secure access to our services while maintaining proper routing based on the requested paths.

module "devlake_alb" {
  source  = "terraform-aws-modules/alb/aws"
  version = "~> 9.13.0"

  name = "devlake-alb"

  load_balancer_type = "application"
  internal           = false
  security_groups    = [aws_security_group.alb.id]
  subnets            = var.public_subnets
  vpc_id             = var.vpc_id

  listeners = {
    https = {
      port            = 443
      protocol        = "HTTPS"
      ssl_policy      = "ELBSecurityPolicy-TLS13-1-2-Res-2021-06"
      certificate_arn = var.certificate_arn
      
      # default action
      forward = {
        target_group_key = "devlake-config-ui-tg"
      }

      rules = {
        grafana = {
          priority = 100
          actions = [{
            type             = "forward"
            target_group_key = "devlake-grafana-tg"
          }]
          conditions = [{
            host_header = {
              values = ["devlake.${var.domain_name}"]
            }
            path_pattern = {
              values = ["/grafana", "/grafana/*"]
            }
          }]
        }
      }
    }
  }

  target_groups = {
    "devlake-grafana-tg" = {
      name        = "devlake-grafana-tg"
      protocol    = "HTTP"
      port        = 3000
      target_type = "ip"
      health_check = {
        enabled             = true
        interval            = 30
        path                = "/api/health"
        port                = "traffic-port"
        timeout             = 5
        healthy_threshold   = 3
        unhealthy_threshold = 3
        matcher             = "200"
      }
      create_attachment = false
    },

    "devlake-config-ui-tg" = {
      name        = "devlake-config-ui-tg"
      protocol    = "HTTP"
      port        = 4000
      target_type = "ip"
      health_check = {
        enabled             = true
        interval            = 30
        path                = "/"
        port                = "traffic-port"
        timeout             = 5
        healthy_threshold   = 3
        unhealthy_threshold = 3
        matcher             = "200"
      }
      create_attachment = false
    }
  }

  tags = {
    Name = var.tag_name
  }
}

DNS Records​

Finally, to use a human readable url, we need to create DNS records in our route53 that points to our alb. For simplicity's sake, I am referencing directly the alb module's outputs, in reality you will probably need to ooutput these from the devlake module itself:

resource "aws_route53_record" "devlake" {
  zone_id = YOUR_DOMAIN_ROUTE53.zone_id
  name    = "devlake.YOUR-DOMAIN.com"
  type    = "A"

  alias {
    name                   = module.devlake_alb.dns_name
    zone_id                = module.devlake_alb.zone_id
    evaluate_target_health = true
  }
}

And thats it, you should be able to access devlake on https://devlake.YOUR-DOMAIN.com!

Cost breakdown

If you want to monitor your costs, first of you need to provide tags with all your resources and second you need to activate the tag on aws so it is registered.

The daily cost is stable on aroun $1.51 a day, which gives about $45 a month. The cost does not include the route53 domain though.

Getting Started​

Before setting up your first Engineering Metrics Dashboard with Apache DevLake, ensure you have the following pre-requisites ready:

• DevLake Version: This feature is available from 1.0.0-beta3 or higher [Installation Guide]
• Repository: Your code should be on GitHub, GitLab, BitBucket, or Azure DevOps. This requirement is just for the initial setup; you can connect other tools later.

Set Up The Workflow​

After installation, connect to your first repository by opening the Config UI or localhost:4000/onboard and you will start the onboarding session as shown in the screenshot below

Step-1: Create Project​

In this step, you'll define a project that aims to enhance your software development process within your organization or team. Remember, the true value lies in the goals behind the metrics rather than the metrics themselves. Focus on setting clear, strategic objectives to ensure that the metrics you track are meaningful and directly contribute to your broader organizational goals.

Step-2: Configure Connection​

• Initiate by creating a Personal Access Token (PAT) on your chosen platform—be it GitHub, GitLab, BitBucket, or Azure DevOps
• This token empowers DevLake to securely interact with the specific repository you're setting up for metric analysis.

Step-3: Add Data Scope​

Adding your data scope is a crucial step where you select the specific repository you want to monitor. Once selected, the dashboard aggregates data pertaining to issues, deployments, pull requests (PRs), workflow runs, GitHub actions, and more, giving you a comprehensive view of your operations. To streamline the onboarding experience and swiftly deliver your initial metrics, the system is configured to collect only the last 14 days of data by default. This ensures a rapid setup so you can start analyzing your metrics almost immediately. However, this setting is flexible— you can adjust the data collection window at any time to suit your project’s needs and objectives.

Metrics Dashboard Overview​

This metrics dashboard provides a clear visualization of the development processes by tracking various key performance indicators from GitHub. The dashboard covers:

• User Requirements (Issues): Monitors new issues and how they are resolved over time, providing insights into the demands and responsiveness of the team.
• Issue Resolution: Shows the number of closed issues and the average time taken for issue resolution, offering a measure of efficiency and agility in problem-solving.
• Pull Requests: Details the number of new pull requests, an indicator of collaboration, alongside the merge time, indicating the pace of code integration.
• Workflow Runs: Highlights the total number of workflow runs and their success rates, illustrating the robustness of the CI/CD pipeline.
• Queue and Processing Times: Analyzes the time taken from issue opening to closure and pull request opening to merge, shedding light on the throughput and process optimization.

By consolidating these metrics, the dashboard serves as a crucial tool for teams to evaluate their development lifecycle and make data-driven improvements.

Next Steps​

Congratulations on setting up your first engineering metrics dashboard using GitHub and Grafana in just a few minutes! As an open-source tool, DevLake supports various data sources and plugins while ensuring that your data remains securely under your control. Learn more about how DevLake can help you by checking the DORA guide, how to customize dashboards, and how to customize data model.

If you haven't already, star DevLake GitHub repository to follow the updates and feel free to post your queries on slack - our vibrant community includes engineering leaders, users, experts, and contributors from diverse backgrounds around the world.

Watch this tutorial on YouTube

In the DevLake Playground, we can explore the data using Jupyter Notebooks. There are some predefined notebooks and you can write your own. A Jupyter notebook combines Python code and documentation, which you can easily customize to your needs with some tweaks. The benefits of these Jupyter notebooks as opposed to Grafana are:

• Grafana is limited to SQL queries for gathering and transforming data, and visualizations for data tables.
• Python (code) offers more flexibility in transforming the data and can easily provide feedback on intermediate steps.
• The playground also supports more visualization types; for example (see the first use case below,) when the data is structured as a graph, we can visualize it with Graphviz.

Use cases​

Analyzing the development process through JIRA statuses​

The DevLake Domain model exposes the changes of issues of our issue tracker, including status changes. If we use that to visualize how issues really flow, we get a rudimentary (automated) value stream map. We can use this to identify bottlenecks in our process or flaws in our process design. This is inspired by this blog post:

For example, in the chart above, we see that it takes on average 15 days for Stories to go from "Ready" to "In Progress". And, it happened 476x within the selected time frame.

And now that we have this data in our playground, we can easily change how we represent it. If we focus on the most common status transitions, we can visualize the distribution of durations in a box plot out of the same data:

This functionality is made available through a predefined notebook, so you can easily run it with your own data.

Explore data across different domains​

Let's say we have a hypothesis: "Defect fixes are more quickly merged than new functionality." Before building a dashboard, we want to determine whether the data quality is good enough and whether we can test this hypothesis. With pandas, we can quickly join different tables from the data model. With the following code, we were able to get a preliminary view:

import pandas as pd

from playground.db_engine import create_db_engine

# the default notebook
DB_URL = "mysql://merico:merico@127.0.0.1:3306/lake"
engine = create_db_engine(DB_URL)

# read tables from database
df_pr_issues = pd.read_sql("select * from pull_request_issues", engine)
df_prs = pd.read_sql("select * from pull_requests", engine)
df_issues = pd.read_sql("select * from issues", engine)

# join pull requests and issues based on rows in pull_request_issues
df = pd.merge(df_pr_issues, df_prs, left_on="pull_request_id", right_on="id", suffixes=('_pr_issues', '_prs'))
df = pd.merge(df, df_issues, left_on="issue_id", right_on="id", suffixes=('_prs', '_issues'))

# set data types correctly
df['created_date_issues'] = pd.to_datetime(df['created_date_issues'])
df['resolution_date'] = pd.to_datetime(df['resolution_date'])
df['created_date_prs'] = pd.to_datetime(df['created_date_prs'])
df['merged_date'] = pd.to_datetime(df['merged_date'])
# calculate lead times
df['issue_lead_time'] = df['resolution_date'] - df['created_date_issues']
df['pr_lead_time'] = df['merged_date'] - df['created_date_prs']
# drop unnecessary columns
df = df[['type_issues', 'title_issues', 'issue_lead_time', 'title_prs', 'pr_lead_time']]

# group lead times by issue_type, add count
df_grouped = df.groupby('type_issues').agg({
    'title_issues': 'count', 
    'issue_lead_time': ['mean', 'median', 'std'], 
    'pr_lead_time': ['mean', 'median', 'std']
})
df.rename(columns={'title_issues': 'issue_count'}, inplace=True)

display(df_grouped)

If we run this example on the Devlake GitHub issues and pull requests (up to March 2024), we get the following output:

Getting started​

We hope you are as excited as we are. We look forward to you joining our community to get your feedback and contributions.

Want to get started? Have a look at the playground repository.

This blog will not aim at a comprehensive summary of the compatibility of database but a record of issues for future reference.

1.Different Data Types​

PostgreSQL does not have a uint type​

type JenkinsBuild struct {
 common.NoPKModel
 JobName           string  `gorm:"primaryKey;type:varchar(255)"`
 Duration          float64 // build time
 DisplayName       string  // "#7"
 EstimatedDuration float64
 Number            int64 `gorm:"primaryKey;type:INT(10) UNSIGNED NOT NULL"`
 Result            string
 Timestamp         int64     // start time
 StartTime         time.Time // convered by timestamp
 CommitSha         string
}

In JenkinsBuild.Number, thegormstruct tag used UNSIGNED, which will lead to the failure to create table and should be removed.

MySQL does not have a bool data type​

For a field defined as bool type in model, gorm will map it to MySQL's TINYINT data type, which can be queried directly with 0 or 1 in SQL, but PostgreSQL has a bool type, so gorm will map it to the BOOL type. If 0 or 1 is still used in SQL to query, there will be a report of error.

Here is an example(only relevant fields are shown in the example). The lookup statement works in MySQL, but will lead to an error in PostgreSQL.

type GitlabMergeRequestNote struct {
 MergeRequestId  int    `gorm:"index"`
 System          bool 
}
 
db.Where("merge_request_id = ? AND `system` = 0", gitlabMr.GitlabId).

After changing the sentence as it follows, an error will still be reported. The reason will be shown in the part about backticks.

db.Where("merge_request_id = ? AND `system` = ?", gitlabMr.GitlabId, false)

2.Different Behaviors​

Bulk insertion​

When ON CONFLIT UPDATE ALL was used to achieve bulk insertion, and if there are multiple records with the same primary key, it will report errors in PostgreSQL but not in MySQL.

Inconsistent definition of model with schema​

For example, in the model definition, GithubPullRequest.AuthorId is of the int type, but this field in the database is of VARCHAR type. When inserting data, MySQL will accept it, but ProstgresSQL will report an error.

type GithubPullRequest struct {
 GithubId        int    `gorm:"primaryKey"`
 RepoId          int    `gorm:"index"`
 Number          int    `gorm:"index"` 
 State           string `gorm:"type:varchar(255)"`
 Title           string `gorm:"type:varchar(255)"`
 GithubCreatedAt time.Time
 GithubUpdatedAt time.Time `gorm:"index"`
 ClosedAt        *time.Time
 // In order to get the following fields, we need to collect PRs individually from GitHub
 Additions      int
 Deletions      int
 Comments       int
 Commits        int
 ReviewComments int
 Merged         bool
 MergedAt       *time.Time
 Body           string
 Type           string `gorm:"type:varchar(255)"`
 Component      string `gorm:"type:varchar(255)"`
 MergeCommitSha string `gorm:"type:varchar(40)"`
 HeadRef        string `gorm:"type:varchar(255)"`
 BaseRef        string `gorm:"type:varchar(255)"`
 BaseCommitSha  string `gorm:"type:varchar(255)"`
 HeadCommitSha  string `gorm:"type:varchar(255)"`
 Url            string `gorm:"type:varchar(255)"`
 AuthorName     string `gorm:"type:varchar(100)"`
 AuthorId       int
 common.NoPKModel
}

3.MySQL-Specific Functions​

We used the GROUP_CONCATfunction in a complex query. Although there are similar functions in PostgreSQL, the function names are different and the usage is slightly different.

cursor2, err := db.Table("pull_requests pr1").
  Joins("left join pull_requests pr2 on pr1.parent_pr_id = pr2.id").Group("pr1.parent_pr_id, pr2.created_date").Where("pr1.parent_pr_id != ''").
  Joins("left join repos on pr2.base_repo_id = repos.id").
  Order("pr2.created_date ASC").
  Select(`pr2.key as parent_pr_key, pr1.parent_pr_id as parent_pr_id, GROUP_CONCAT(pr1.base_ref order by pr1.base_ref ASC) as cherrypick_base_branches, 
   GROUP_CONCAT(pr1.key order by pr1.base_ref ASC) as cherrypick_pr_keys, repos.name as repo_name, 
   concat(repos.url, '/pull/', pr2.key) as parent_pr_url`).Rows()

Solution: We finally decided to use two steps to achieve the GROUP_CONCAT function. First we used the simplest SQL query to get multiple pieces of the sorted data, and then used the code to group them.

After modification:

    cursor2, err := db.Raw(
  `
   SELECT pr2.pull_request_key                 AS parent_pr_key,
          pr1.parent_pr_id                     AS parent_pr_id,
          pr1.base_ref                         AS cherrypick_base_branch,
          pr1.pull_request_key                 AS cherrypick_pr_key,
          repos.NAME                           AS repo_name,
          Concat(repos.url, '/pull/', pr2.pull_request_key) AS parent_pr_url,
        pr2.created_date
   FROM   pull_requests pr1
          LEFT JOIN pull_requests pr2
                 ON pr1.parent_pr_id = pr2.id
          LEFT JOIN repos
                 ON pr2.base_repo_id = repos.id
   WHERE  pr1.parent_pr_id != ''
   ORDER  BY pr1.parent_pr_id,
             pr2.created_date,
       pr1.base_ref ASC
   `).Rows()

4.Different Grammar​

Backticks​

We used backticks in some SQL statements to protect field names from conflicting with MySQL reserved words, which can lead to errors in PostgreSQL. To solve this problem we revisited our code, modified all field names that conflict with reserved words, and removed the backticks in the SQL statement. In the example just mentioned:

db.Where("merge_request_id = ? AND `system` = ?", gitlabMr.GitlabId, false)

Solution: We changed system to is_system to avoid the usage of backticks.

db.Where("merge_request_id = ? AND is_system = ?", gitlabMr.GitlabId, false)

Non-standard delete statement​

There were delete statements as followed in our code, which are legal in MySQL but will report an error in PostgreSQL.

err := db.Exec(`
 DELETE ic
 FROM jira_issue_commits ic
 LEFT JOIN jira_board_issues bi ON (bi.source_id = ic.source_id AND bi.issue_id = ic.issue_id)
 WHERE ic.source_id = ? AND bi.board_id = ?
 `, sourceId, boardId).Error

Apache DevLack Architecture Overview​

• The left side of the following screenshot is an integrative DevOps data plugin, the existing plugins include Github, GitLab, JIRA, Jenkins, Tapd, Feishu, and the most featured analysis engine in the Simayi platform.
• The main framework in the middle of the following screenshot, completes data collection, expansion, and conversion to the domain layer by running subtasks in the plugins. The user can trigger the tasks by config-UI or all API.
• RMDBS currently supports Mysql and PostgresSQL, more databases will be supported in the future.
• Grafana can generate different types of needed data by using SQL.

Then let’s move on to how to start running DevLake.

Start the system​

Before the Golang program runs, it will automatically call the init() method in the package. We need to focus on the loading of the services package. The following code has detailed comments:

func init() {
var err error
// get initial config information
cfg = config.GetConfig()
// get Database
db, err = runner.NewGormDb(cfg, logger.Global.Nested("db"))
// configure time zone
location := cron.WithLocation(time.UTC)
// create scheduled task manager
cronManager = cron.New(location)
if err != nil {
panic(err)
}
// initialize the data migration
migration.Init(db)
// register the framework's data migration scripts
migrationscripts.RegisterAll()
// load plugin, loads all .so files in the folder cfg.GetString("PLUGIN_DIR")，in th LoadPlugins method()，specifically, LoadPlugins stores the pluginName:PluginMeta key-value pair into core.plugins by calling runner.
err = runner.LoadPlugins(
cfg.GetString("PLUGIN_DIR"),
cfg,
logger.Global.Nested("plugin"),
db,
)
if err != nil {
panic(err)
}
// run data migration scripts to complete the initializztion work of tables in the databse framework layer.
err = migration.Execute(context.Background())
if err != nil {
panic(err)
}

// call service init
pipelineServiceInit()
}

The execution principle of DevLake​

The running process of the pipeline Before we go through the pipeline process, we need to know the Blueprint first.

Blueprint is a timed task that contains all the subtasks and plans that need to be executed. Each execution record of Blueprint is a historical run, AKA Pipeline. Which presents a trigger for DevLack to complete one or more data collection transformation tasks through one or more plugins.

The following is the pipeline running flow chart.

A pipeline contains a two-dimensional array of tasks, mainly to ensure that a series of tasks are executed in a preset order. Like the following screenshot if the plugin of Stage 3 needs to rely on the other plugin to prepare the data(eg: the operation of refdiff needs to rely on gitextractor and Github, for more information on data sources and plugins, please refer to the documentation, then when Stage 3 starts to execute, it needs to ensure that its dependencies have been executed in Stage 1 and Stage 2.

Task running process

The plugin tasks in stage1, stage2, and stage3 are executed in parallel:

The next step is to execute the subtasks in the plugin sequentially.

1. The work before the RunTask is to prepare the params for the RunTask method to call, like logger, db, context and etc.
2. The main method of RunTask is mainly to update the tasks in the Database, at the same time prepare to run the options of the plugins task.
3. RunpluginTask will obtain the corresponding PluginMeta through core.Getplugin(pluginName), then obtains the PluginTask via PluginMeta, and then executes RunPluginSubTasks.

The running process of each plugin subtask(the relevant interface and func will be explained in the next section)

1. Get all available subtasks subtaskMeta of all th4e plugins by calling SubTaskMetas().
2. Use options[‘task’] and subtaskMeta to form a set of subtasks to be executed subtaskMetas.
3. Calculate how many subtasks in total
4. Build taskCtx via helper.NewDefaultTaskContext.
5. Build taskData via call pluginTask.PrepareTaskData.
6. Iterate over all subtasks in subtaskMetas.
   1. Get subtaskCtx of subtask via call taskCtx.SubTaskContext(subtaskMeta.Name).
   2. Run subtaskMeta.EntryPoint(subtaskCtx)

Important interfaces in DevLake​

1. PluginMeta: Contains the two basic methods of plugins, which all plugins need to implement. And stored in core.plugins when the system starts. And obtained through core.GetPlugin when executing plugin tasks.

type PluginMeta interface {
   Description() string
   //PkgPath information will be lost when compiled as plugin(.so), this func will return that info
   RootPkgPath() string
}

2. PluginTask: It can be obtained by PluginMeta, after the plugin implemented this method, Framework can run the subtask directly, instead of letting the plugin itself run it, the biggest benefit of this is that the subtasks of the plugin are easier to implement, and we can more easily leverage(such as adding logs, etc.) during the operation of the plugin.

type PluginTask interface {
   // return all available subtasks, framework will run them for you in order
   SubTaskMetas() []SubTaskMeta
   // based on task context and user input options, return data that shared among all subtasks
   PrepareTaskData(taskCtx TaskContext, options map[string]interface{}) (interface{}, error)
}

3. Every plugin has a taskData, which contains configuration options, apiClient, and other properties of plugins.(like the github has repo information)
4. SubTaskMeta:: the meta data of a subtask, every subtask will define a SubTaskMeta.

var CollectMeetingTopUserItemMeta = core.SubTaskMeta{
   Name: "collectMeetingTopUserItem",
   EntryPoint: CollectMeetingTopUserItem,
   EnabledByDefault: true,
   Description: "Collect top user meeting data from Feishu api",
}

5. ExecContext: defines all resources needed to execute (sub)tasks.
6. SubTaskContext: defines all resources need to execute subtask(including ExecContext).
7. TaskContext: defines all resources need to execute tasks(including ExecContext). The difference with SubTaskContext is the TaskContext() method in SubTaskContext can retire TaskContext, but SubTaskContext(subtask string) method in TaskContext can return SubTaskContext, which means the subtask belongs to the plugin task, so we use the different contexts to distinguish this.
8. SubTaskEntryPoint: all the subtasks in the plugin have to implement this function so that they can be coordinated and arranged by the framework layer.

Further Plan​

This blog introduced the basics of the DevLack framework and how it starts and runs, there are 3 more contexts api_collector, api_extractor, and data_convertor will be explained in the next blog.

What is DevLake?​

Launched in December of 2021, Apache DevLake is an open-source dev data platform that ingests, analyzes, and visualizes the fragmented data in developer tools.

Software development is complex, requiring many tools and processes, and as a result creates a storm of data scattered across tools in many formats. This makes it difficult to organize, query, and make sense of. We built Apache DevLake, to make it easy to make sense of this rich data and to translate it into actionable insights.

Apache DevLake's key features:​

• DevOps data collection across software development lifecycle (SDLC) to connect data islands
• Standardized data models with out-of-the-box metrics and customizable dashboards
• Flexible plugin system for user-defined data integration and transformation

Below is the architecture of Apache DevLake:

Why join Apache Incubator?​

First and foremost, we firmly believe in Apache's principle of "Community over Code" and "The Apache Way". We look forward to building a vibrant, inclusive, and diverse community under the guidance of our mentors through the incubator journey.

We also feel that the Apache community has one of the strongest ecosystems when it comes to data-oriented open-source projects.

Last and not least, the support and enthusiasm of the Apache community and mentors made it clear to us that this would be a place where we can truly evolve and nurture Apache DevLake.

In the months leading up to DevLake's acceptance, our mentors provided guidance and suggestions instrumental to making this important moment a reality. We want to take this opportunity to thank and acknowledge them:

• Willem Ning Jiang: Apache DevLake Champion; ASF Member & Board Director

• Liang Zhang: Founder & CEO of SphereEx; ASF Member, Founder & PMC Chair of Apache ShardingSphere

• Lidong Dai: ASF Member; Apache DolphinScheduler PMC Chair

• Sijie Guo: ASF Member, PMC member of Apache Pulsar; Founder & CEO of StreamNative

• Felix Cheung: ASF Member, PMC on Spark, Superset, Yunikorn, Zeppelin, Pinot, and Incubator. SVP of Engineering at SafeGraph.

• Jean-Baptiste Onofré: ASF Member, Karaf PMC Chair, PMC on ActiveMQ, Archiva, Aries, Beam, Brooklyn, Camel, Carbondata, Felix, Incubator, and many more.

Apache DevLake Future Roadmap​

• Enhance system scalability and performance in large-scale data scenarios.
• Integrate more data sources and tools (JIRA, GitHub, GitLab, and Jenkins are already supported.)
• Enable support for OLAP databases, providing users with more choices.
• Provide more scenario-specific, out-of-the-box dashboards and templates reflecting best practices and well-known methodologies to improve usability.

Join us!​

We invite developers and those passionate about data-driven engineering to 'dive into the lake' with us, and welcome contributions of all kinds. Join us on Slack and at our weekly open source community meetups🥳

Apache DevLake (Incubating) Links:

• GitHub: https://github.com/apache/incubator-devlake
• Official Website: https://devlake.apache.org/
• Slack: https://join.slack.com/t/devlake-io/shared_invite/zt-1lkgbdmys-AU2azidzO1u~mtjlg9my7A
• Podling Website：https://incubator.apache.org/projects/devlake.html