Skip to main content
Version: v0.18

Summary

For some metric calculations such as the DORA metric, we often encounter situations requiring comprehensive calculations based on data from multiple data sources.

For example, we may use GitLab for code hosting, Jenkins for CI/CD, to calculate PR deployment cycle time, we need to know which GitLab Projects and Jenkins Jobs are related for correctness and performance reasons.

However, in most cases, we have multiple GitLab Projects / Jenkins Jobs that belong to different teams in our Apache DevLake database.

To distinguish them into different groups. The Project is introduced in v0.15. Essentially, a project consists of a set of Scopes, i.e., a couple of GitLab Projects, Jira Boards or Jenkins Jobs, etc.

Project is a set of Scope from different domains, a way to group different resources, and it is crucial for some metric calculation like Dora.

Next, let us introduce Project in the following order:

  • Project related models
  • Related APIs that can be used to manipulate Project models
  • The interface that needs to be implemented when developing various plugins to support the Project.
    • The interface that needs to be implemented to develop the Data Source Plugin
    • The interface that needs to be implemented to develop the Metric Plugins

Models

To support project we contains the following three models:

  • projects describes a project object, including its name, creation and update time and other basic information
  • project_metric_settings describes what metric plugins a project had enabled.
  • project_mapping describes the mapping relationship of project and scope, including the name of the project、the table name of Scope and the row_id in the Scope table.

projects

fieldtypelengthdescriptionkey
namevarchar255name for projectPK
descriptionlongtextdescription of the project
created_atdatetime3created time of project
updated_atdatetime3last updated time of project

example

namedescribecreated_atupdated_at
project_1this is one of the test projects2022-11-01 01:22:13.0002022-11-01 02:24:15.000
project_2this is another project test project2022-11-01 01:23:29.0002022-11-01 02:27:24.000

project_metric_settings

fieldtypelengthdescriptionkey
project_namevarchar255name for projectPK
plugin_namevarchar255name for pluginPK
plugin_optionlongtextcheck if metric plugins have been enabled by the project
enabletinyint1if the metric plugins is enabled

example

project_nameplugin_nameplugin_optionenable
project_1dora{}true
project_2dora{}false

project_mapping

fieldtypelengthdescriptionkey
project_namevarchar255name for projectPK
tablevarchar255the table name of ScopePK
row_idvarchar255the row_id in the Scope tablePK

example

project_nametablerow_id
project_1Repogitlab:GithubRepo:1:lake
project_1Boardjira:JiraBoard:1:lake
project_2Repogithub:GithubRepo:1:lake

How to manage project via API

For API specification, please check the swagger doc(by visiting [Your Config-UI Host]/api/swagger/index.html). Related endpoints:

  1. /projects
  2. /projects/:projectName/metrics
  3. /plugins

The interface that needs to be implemented

We divide plugins into two categories

  • The first category is Data Source Plugin, such as GitLab GitHub Jira Jenkins, etc. These plugins collect data from various data sources
  • The second category is Metric Plugin, such as Dora, etc. These plugins do not directly contact the data source but do secondary calculations based on the collected data after the Data Source Plugin works

Data Source Plugin

For example GitLab GitHub Jira Jenkins etc.

These plugins, from various data sources, extract data into the database and store them, they deal directly with the data source, so we classify them as Data Source Plugin.

the DataSourcePluginBlueprintV200 interface

Data Source Plugin needs to implement DataSourcePluginBlueprintV200 interface to support project

The interface definition for this interface is as follows

// DataSourcePluginBlueprintV200 extends the V100 to provide support for
// Project, so that complex metrics like DORA can be implemented based on a set
// of Data Scopes
type DataSourcePluginBlueprintV200 interface {
MakeDataSourcePipelinePlanV200(
connectionId uint64,
scopes []*BlueprintScopeV200,
syncPolicy BlueprintSyncPolicy,
) (PipelinePlan, []Scope, errors.Error)
}

scopes in input parameters is a set of arrays containing IDs, Names, and Entities.

The input data format is as follows:

[]*core.BlueprintScopeV200{
{
Entities: []string{"CODE", "TICKET", "CICD"},
Id: "37",
Name: "test",
},
}

syncPolicy in input parameters contains some option settings, whose structure is defined as follows:

type BlueprintSyncPolicy struct {
Version string `json:"version" validate:"required,semver,oneof=1.0.0"`
SkipOnFail bool `json:"skipOnFail"`
CreatedDateAfter *time.Time `json:"createdDateAfter"`
}

PipelinePlan in output is a part of blueprint JSON:

The input data format is as follows:(Take GitLab plugin as an example)

core.PipelinePlan{
{
{
Plugin: "gitlab",
Subtasks: []string{
tasks.ConvertProjectMeta.Name,
tasks.CollectApiIssuesMeta.Name,
tasks.ExtractApiIssuesMeta.Name,
tasks.ConvertIssuesMeta.Name,
tasks.ConvertIssueLabelsMeta.Name,
tasks.CollectApiJobsMeta.Name,
tasks.ExtractApiJobsMeta.Name,
tasks.CollectApiPipelinesMeta.Name,
tasks.ExtractApiPipelinesMeta.Name,
},
Options: map[string]interface{}{
"connectionId": uint64(1),
"projectId": testID,
},
},
{
Plugin: "gitextractor",
Options: map[string]interface{}{
"proxy": "",
"repoId": repoId,
"url": "https://git:nddtf@this_is_cloneUrl",
},
},
},
{
{
Plugin: "refdiff",
Options: map[string]interface{}{
"tagsLimit": 10,
"tagsOrder": "reverse semver",
"tagsPattern": "pattern",
},
},
},
}

project needs to provide a specific set of Scopes for a specific connection to the plugin through this interface, and then obtain the plugin involved in the PipelineTask All plugins and corresponding parameter information. At the same time, the plugin needs to convert entities like repo and board in the data source into a scope interface that project can understand

The corresponding scope interface has been implemented at following files of in the framework layer:

  • models/domainlayer/devops/cicd_scope.go
  • models/domainlayer/ticket/board.go
  • models/domainlayer/code/repo.go

In the plugins/gitlab/impl/impl.go file, there is a GitLab plugin implementation of the above interface, which can be used as a reference.

And the plugins/gitlab/api/blueprint_v200.go contains implementation details.

The following files contain the models that the relevant implementations depend on for reference:

  • plugins/gitlab/models/project.go
  • plugins/gitlab/models/scope_config.go

Metric Plugins

For example Dora, and Refdff plugins belong to the Metric Plugins

These plugins are mainly for calculating various metrics, they do not directly contact the data source, so we classify them as Metric Plugins.

The PluginMetric Interface

Metric Plugins needs to implement the PluginMetric interface to support project

The interface definition for this interface looks like this:

type PluginMetric interface {
// returns a list of required data entities and expected features.
// [{ "model": "cicd_tasks", "requiredFields": {"column": "type", "execptedValue": "Deployment"}}, ...]
RequiredDataEntities() (data []map[string]interface{}, err errors.Error)

// returns if the metric depends on Project for calculation.
// Currently, only dora would return true.
IsProjectMetric() bool

// indicates which plugins must be executed before executing this one.
// declare a set of dependencies with this
RunAfter() ([]string, errors.Error)

// returns an empty pointer of the plugin setting struct.
// (no concrete usage at this point)
Settings() (p interface{})
}

Project needs PluginMetric to know whether a Metric Plugin is dependent on project, and the tables and fields required in its calculation process.

In the plugins/dora/impl/impl.go file, there is a Dora plugin implementation of the above interface, which can be used as a sample reference.You can find it by searching the following fields:

  • func (plugin Dora) RequiredDataEntities() (data []map[string]interface{}, err errors.Error)
  • func (plugin Dora) IsProjectMetric() bool
  • func (plugin Dora) RunAfter() ([]string, errors.Error)
  • func (plugin Dora) Settings() interface{}

References

To dig deeper into developing and utilizing our built-in functions and have a better developer experience, feel free to dive into our godoc reference.