Supported Data Sources

Data Sources and Data Plugins

Apache DevLake(incubating) supports the following data sources. The data from each data source is collected with one or more plugins. Detailed plugin docs can be found here.

Data Source	Domain(s)	Supported Versions	Config UI Availability	Triggered Plugins	Collection Mode
GitHub	Source Code Management, Code Review, Issue Tracking, CI/CD (GitHub Actions)	Cloud	Available	github, gitextractor	Incremental Sync
GitLab	Source Code Management, Code Review, Issue Tracking, CI/CD (GitLab CI)	Cloud, Community Edition 11+	Available	gitlab, gitextractor	Full Refresh, Incremental Sync(for issues,MRs)
Jira	Issue Tracking	Cloud, Server/Data Center 7.x, 8.x	Available	jira	Full Refresh, Incremental Sync(for issues and related)
Jenkins	CI/CD	2.263.x+	Available	jenkins	Incremental Sync
BitBucket (Beta)	Source Code Management, Code Review	Cloud	Advanced Mode Available	bitbucket, gitextractor	Full Refresh
TAPD (Beta)	Issue Tracking	Cloud	Advanced Mode Available	tapd	Full Refresh, Incremental Sync(for stories, bugs, tasks)
Zentao (Beta)	Issue Tracking	v17.x, v18.x	Advanced Mode Available	zentao	Full Refresh
Gitee (WIP)	Source Code Management, Code Review, Issue Tracking	Cloud	Not Available	gitee, gitextractor	Full Refresh, Incremental Sync(for issues,MRs)
PagerDuty (WIP)	Issue Tracking	Cloud	Not Available	pagerduty	Full Refresh
Feishu (WIP)	Calendar	Cloud	Not Available	feishu	Full Refresh
AE	Source Code Management	On-prem	Not Available	ae	Full Refresh
Sonarqube	CODE QUALITY	SonarQube v8.x, v9.x	Available	sonarqube	Full Refresh
Bamboo CI(WIP)	CI/CD	v6.8.1	Not Available	bamboo	Full Refresh
Azure Devops (Beta)	CI/CD, Source Code Management, Code Review	Cloud	Available	azuredevops, gitextractor	Full Refresh

Data Collection Scope By Each Plugin

This table shows the entities collected by each plugin. Domain layer entities in this table are consistent with the entities here. ✅ : Collect by default. 💪 : Collect not by default. You need to add the corresponding subtasks to collect these entities in the advanced mode.

Domain Layer Entities	ae	dora	gitextractor	incoming webhook	github	gitlab	jenkins	jira	refdiff	tapd	sonarqube	bamboo	azuredevops
accounts					✅	✅		✅		✅	✅
board_issues					✅	✅		✅		✅
board_repos					✅	✅		✅
board_sprints					✅			✅		✅
boards					✅	✅		✅		✅
cicd_pipeline_commits		✅			✅	✅	✅					✅	✅
cicd_pipelines		✅			✅	✅	✅					✅	✅
cicd_scopes		✅			✅	✅	✅					✅	✅
cicd_tasks		✅		💪	✅	✅	✅					✅	✅
commit_file_components			✅
commit_files			✅
commit_line_change			✅
commit_parents			✅
commits	✅		✅		💪	💪
commits_diffs									✅
components
finished_commits_diffs
issue_changelogs								✅		✅
issue_comments					✅					✅
issue_commits								✅		✅
issue_labels					✅	✅				✅
issue_repo_commits								✅
issue_worklogs								✅		✅
issues					✅			✅		✅
project_issue_metrics		✅			✅	✅		✅		✅
project_mapping		✅			✅	✅	✅	✅		✅
project_metrics		✅			✅	✅	✅	✅		✅
project_pr_metrics		✅			✅	✅				✅
projects		✅			✅	✅	✅	✅		✅
pull_request_comments					✅	✅
pull_request_commits					✅	✅							✅
pull_request_issues					✅
pull_request_labels					✅	✅
pull_requests					✅	✅							✅
ref_commits									✅
refs			✅						✅
refs_issues_diffs									✅
ref_pr_cherry_picks									✅
repo_commits			✅		💪	💪
repo_snapshot			✅
repos					✅	✅
sprint_issues					✅			✅		✅
sprints					✅			✅		✅
team_users
teams
user_account
users								✅		✅
cq_projects								✅		✅	✅
cq_issues								✅		✅	✅
cq_issue_code_blocks								✅		✅	✅
cq_file_metrics								✅		✅	✅

Data Sync Policy

bold: means it may collect slowly.

*bold*: means it may collect very slowly.

Jira

Subtask Name	Estimated Max Number of Request	Does It support Incremental Collection?	Does It Support Time Filter?
CollectStatusMeta	1	-	-
CollectProjectsMeta	<10	❌	-
CollectIssueTypesMeta	<10	❌	-
CollectIssuesMeta	<10^4	✅	✅
CollectIssueChangelogsMeta	1000~10^5	✅	✅
CollectAccountsMeta	<10^3	❌	❌
CollectWorklogsMeta	1000~10^5	✅	✅
CollectRemotelinksMeta	1000~10^5	✅	✅
CollectSprintsMeta	<100	❌	❌
CollectEpicsMeta	<100	❌	✅

Jenkins

Subtask Name	Estimated Max Number of Request	Does It support Incremental Collection?	Does It Support Time Filter?
CollectApiBuildsMeta	≈100	❌	❌
CollectApiStagesMeta	≈10^4	❌	✅

Gitlab

Subtask Name	Estimated Max Number of Request	Does It support Incremental Collection?	Does It Support Time Filter?
CollectApiIssuesMeta	<10^4	✅	✅
CollectApiMergeRequestsMeta	<10^3	✅	✅
CollectApiMrNotesMeta	<10^5	❌	✅
CollectApiMrCommitsMeta	<10^5	❌	✅
CollectApiPipelinesMeta	<10^4	✅	❌
CollectApiJobsMeta	<10^5	❌	✅

Github

Subtask Name	Estimated Max Number of Request	Does It support Incremental Collection?	Does It Support Time Filter?
---------------------------------	Common	-----------------------
CollectMilestonesMeta	≈10	✅	❌
CollectRunsMeta	<10^4	✅	✅
CollectApiCommentsMeta	400 (max page that GitHub supports)	✅	✅
CollectApiEventsMeta	400 (max page that GitHub supports)	❌	❌
CollectApiPullRequestReviewsMeta	<10^5	✅	✅
---------------------------------	Graphql Only (Default)	-----------------------
CollectIssueMeta	≈10^4	❌	✅
CollectPrMeta	≈10^3	❌	✅
CollectCheckRunMeta	<10^4	❌	✅
CollectAccountMeta	≈10^2	❌	-
---------------------------------	Restful Only (Not by Default)	-----------------------
CollectApiIssuesMeta	≈10^4	✅	❌
CollectApiPullRequestsMeta	≈10^2	❌	❌
CollectApiPullRequestCommitsMeta	≈10^4	✅	✅
CollectApiPrReviewCommentsMeta	≈10^4	✅	✅
CollectAccountsMeta	≈10^4	❌	❌
CollectAccountOrgMeta	≈10^4	❌	❌
CollectJobsMeta	<10^6	❌	✅
CollectApiCommitsMeta	Not enabled	-	-
CollectApiCommitStatsMeta	Not enabled	-	-

Feishu

Subtask Name	Estimated Max Number of Request	Does It support Incremental Collection?	Does It Support Time Filter?
CollectMeetingTopUserItemMeta	≈10^3	❌	✅

Bitbucket

Subtask Name	Estimated Max Number of Request	Does It support Incremental Collection?	Does It Support Time Filter?
CollectApiRepoMeta	1	❌	❌
CollectApiPullRequestsMeta	≈10^3	❌	❌
CollectApiIssuesMeta	≈10^4	❌	❌
CollectApiPrCommentsMeta	≈10^5	❌	❌
*CollectApiIssueCommentsMeta*	≈10^6	❌	❌
CollectApiPipelinesMeta	<10^4	❌	❌
CollectApiDeploymentsMeta	<10^2	❌	❌

Gitee

Subtask Name	Estimated Max Number of Request	Does It support Incremental Collection?	Does It Support Time Filter?
CollectApiRepoMeta	1	❌	❌
CollectApiPullRequestsMeta	≈10^3	✅	❌
CollectApiIssuesMeta	≈10^4	✅	❌
CollectCommitsMeta?	≈10^4	✅	❌
CollectApiPrCommentsMeta	≈10^5	❌	❌
*CollectApiIssueCommentsMeta*	≈10^6	✅	❌
CollectApiPullRequestCommitsMeta	≈10^5	❌	❌
CollectApiPullRequestReviewsMeta	≈10^5	❌	❌
*CollectApiCommitStatsMeta*	≈10^6 (Not enable)	❌	❌

SonarQube

Subtask Name	Estimated Max Number of Request	Does It support Incremental Collection?	Does It Support Time Filter?
CollectAccounts	<10^4	❌	❌
CollectIssues	<10^4	❌	❌
CollectHotspots	<10^4	❌	❌
CollectFilemetrics	<10^4	❌	❌
CollectAdditionalFilemetrics	<10^4	❌	❌

Bamboo

Subtask Name	Estimated Max Number of Request	Does It support Incremental Collection?	Does It Support Time Filter?
CollectPlanMeta	<10^4	❌	❌
CollectJobMeta	<10^5	❌	❌
CollectPlanBuildMeta	<10^6	❌	❌
CollectJobBuildMeta	<10^6	❌	❌
CollectDeployMeta	1	❌	❌

Zentao

Subtask Name	Estimated Max Number of Request	Does It support Incremental Collection?	Does It Support Time Filter?
CollectExecutionMeta	<10^3	❌	❌
CollectStoryMeta	<10^4	❌	❌
CollectBugMeta	<10^4	❌	❌
CollectTaskMeta	<10^4	❌	❌
CollectAccountMeta	≈10^2	❌	❌
CollectDepartmentMeta	≈10^2	❌	❌

Tapd

Subtask Name	Estimated Max Number of Request	Does It support Incremental Collection?	Does It Support Time Filter?
CollectWorkitemTypesMeta	≈10	-	-
CollectStoryCustomFieldsMeta	≈10	-	-
CollectTaskCustomFieldsMeta	≈10	-	-
CollectBugCustomFieldsMeta	≈10	-	-
CollectStoryCategoriesMeta	≈10	-	-
CollectStoryStatusMeta	≈10	-	-
CollectStoryStatusLastStepMeta	≈10	-	-
CollectBugStatusMeta	≈10	-	-
CollectBugStatusLastStepMeta	≈10	-	-
CollectAccountsMeta	≈10^3	❌	❌
CollectIterationMeta	≈10^4	✅	✅
CollectStoryMeta	≈10^4	✅	✅
CollectBugMeta	≈10^4	✅	✅
CollectTaskMeta	≈10^4	✅	✅
CollectBugChangelogMeta	≈10^6	✅	✅
CollectStoryChangelogMeta	≈10^6	✅	✅
CollectTaskChangelogMeta	≈10^6	✅	✅
CollectWorklogMeta	≈10^6	✅	✅
CollectBugCommitMeta	≈10^6	✅	✅
CollectStoryCommitMeta	≈10^6	✅	✅
CollectTaskCommitMeta	≈10^6	✅	✅
CollectStoryBugMeta	≈10^6	✅	✅

Azure Devops

Subtask Name	Estimated Max Number of Request	Does It support Incremental Collection?	Does It Support Time Filter?
CollectBuilds	<10^3	❌	❌
CollectJobs	<10^4	❌	❌
CollectPullRequests	<10^3	❌	❌
CollectPullRequestCommits	<10^4	❌	❌