Gitextractor Plugin#
The gitextractor plugin extracts Git history data (commits, branches, tags, diff lines) from remote repositories and stores them in DevLake's domain layer. It is registered as a standard DevLake plugin implementing PluginMeta, PluginTask, and PluginModel .
Subtask Pipeline#
The plugin runs five ordered subtasks :
| Subtask | Purpose |
|---|---|
Clone Git Repo | Clone the repo into a temp dir and initialize the collector |
CollectGitCommit | Walk commits, store stats and parent relationships |
CollectGitBranch | Enumerate all local and remote branches |
CollectGitTag | Enumerate all tags |
CollectGitDiffLine | Build a per-line blame snapshot |
Configuration Options#
PrepareTaskData in impl.go decodes per-task options and applies .env fallbacks for three key flags:
| Option | Env key | Default |
|---|---|---|
UseGoGit | UseGoGit | false (use libgit2) |
SkipCommitStat | SKIP_COMMIT_STAT | false |
SkipCommitFiles | SKIP_COMMIT_FILES | true |
SkipCommitFiles is true by default because collecting per-file diff data is expensive .
Repository Cloning (CloneGitRepo subtask)#
tasks/repo_cloner.go contains the CloneGitRepo entry point. Its responsibilities are:
- Allocate a temp dir via
os.MkdirTemp("", "gitextractor"). - Clone using
parser.NewGitcliClonerβ the shellgitbinary is always used for cloning; the Go library is only used for reading . - Open the cloned repo with either
NewLibgit2RepoCollector(default) orNewGogitRepoCollector. - Register a cleanup callback via
repoCollector.SetCleanUp(cleanup)and store the collector intaskData.GitRepofor downstream subtasks .
Shallow Clone Strategy#
clone_gitcli.go implements three cloning modes selected at runtime by CloneRepo :
- Full clone (
fullClone) β used on first run (nosincetimestamp). Runsgit clone --bare. - Shallow clone (
shallowClone) β used for incremental runs when the remote supports shallow clones. Clones with--depth=1, reconfigures fetch refspecs to cover all branches, then fetches--shallow-since=<timestamp>. - Double clone (
doubleClone) β used when the remote does not support shallow clones (NoShallowClone: true). Full-clones into a second temp dir (os.MkdirTemp("", "gitextint")), then performs a local shallow clone from that intermediary usingfile://. This reduces libgit2 memory usage on large repos.
After shallow/double clone, deepen() runs git repack -d and git fetch --deepen=1 to avoid a known object-unshallow error on merge commits .
The CloneRepoConfig struct is tracked by the SubtaskStateManager; a change to any of its fields forces a full re-sync.
Cleanup Pattern#
Cleanup is injected as a plain func() callback rather than being wired directly into the collector at construction time. This keeps the RepoCollector interface agnostic of how the temp directory was created.
The cleanup closure calls os.RemoveAll(localDir) to delete the temp directory and repoCloner.CloseRepo() to persist the state manager (which records the since timestamp for the next incremental run).
Both collector implementations store the callback in a cleanUp/cleanup field and invoke it inside Close():
GogitRepoCollector.Closeβ closes the store, then callsr.cleanUp()if set .Libgit2RepoCollector.Closeβ defersr.cleanup()and closes the store .
Plugin-level Close (impl.go) is called by the framework at task teardown. It reads GIT_EXTRACTOR_KEEP_REPO from config β if unset, it calls taskData.GitRepo.Close(ctx), which triggers the cleanup callback and removes the temp dir .
Note: Set
GIT_EXTRACTOR_KEEP_REPO=truein.envto retain the cloned directory for debugging.
Repo Collector Backends#
Two backend implementations share the RepoCollector interface:
| Backend | File | Library | Notes |
|---|---|---|---|
| libgit2 (default) | parser/repo_libgit2.go | git2go/v33 | Better performance per in-code benchmarks |
| go-git | parser/repo_gogit.go | go-git/v5 | Enabled via UseGoGit=true |
Both implement SetCleanUp(f func()) error, Close(ctx) error, and methods for collecting tags, branches, commits, and diff lines.
Shallow-clone edge case: Both collectors skip computing commit statistics when a commit has parents that are not present in the ODB (i.e., the boundary of a shallow clone), to avoid writing inflated addition counts to the database .
Key Files#
| File | Role |
|---|---|
backend/plugins/gitextractor/impl/impl.go | Plugin entry point, PrepareTaskData, Close |
backend/plugins/gitextractor/tasks/repo_cloner.go | CloneGitRepo subtask β temp dir allocation, cloner + collector wiring |
backend/plugins/gitextractor/parser/clone_gitcli.go | GitcliCloner β full/shallow/double clone logic |
backend/plugins/gitextractor/parser/repo_libgit2.go | libgit2-backed collector (default) |
backend/plugins/gitextractor/parser/repo_gogit.go | go-git-backed collector |