Gitextractor Plugin#

The gitextractor plugin extracts Git history data (commits, branches, tags, diff lines) from remote repositories and stores them in DevLake's domain layer. It is registered as a standard DevLake plugin implementing PluginMeta, PluginTask, and PluginModel 1.

Subtask Pipeline#

The plugin runs five ordered subtasks 2:

Configuration Options#

PrepareTaskData in impl.go decodes per-task options and applies .env fallbacks for three key flags:

SkipCommitFiles is true by default because collecting per-file diff data is expensive 3.

Repository Cloning (CloneGitRepo subtask)#

tasks/repo_cloner.go contains the CloneGitRepo entry point. Its responsibilities are:

1. Allocate a temp dir via os.MkdirTemp("", "gitextractor") 4.
2. Clone using parser.NewGitcliCloner — the shell git binary is always used for cloning; the Go library is only used for reading 5.
3. Open the cloned repo with either NewLibgit2RepoCollector (default) or NewGogitRepoCollector 6.
4. Register a cleanup callback via repoCollector.SetCleanUp(cleanup) and store the collector in taskData.GitRepo for downstream subtasks 7.

Shallow Clone Strategy#

clone_gitcli.go implements three cloning modes selected at runtime by CloneRepo 8:

• Full clone (fullClone) — used on first run (no since timestamp). Runs git clone --bare 9.
• Shallow clone (shallowClone) — used for incremental runs when the remote supports shallow clones. Clones with --depth=1, reconfigures fetch refspecs to cover all branches, then fetches --shallow-since=<timestamp> 10.
• Double clone (doubleClone) — used when the remote does not support shallow clones (NoShallowClone: true). Full-clones into a second temp dir (os.MkdirTemp("", "gitextint")), then performs a local shallow clone from that intermediary using file:// 11. This reduces libgit2 memory usage on large repos.

After shallow/double clone, deepen() runs git repack -d and git fetch --deepen=1 to avoid a known object-unshallow error on merge commits 12.

The CloneRepoConfig struct 13 is tracked by the SubtaskStateManager; a change to any of its fields forces a full re-sync.

Cleanup Pattern#

Cleanup is injected as a plain func() callback rather than being wired directly into the collector at construction time. This keeps the RepoCollector interface agnostic of how the temp directory was created.

The cleanup closure calls os.RemoveAll(localDir) to delete the temp directory and repoCloner.CloseRepo() to persist the state manager (which records the since timestamp for the next incremental run).

Both collector implementations store the callback in a cleanUp/cleanup field and invoke it inside Close():

• GogitRepoCollector.Close — closes the store, then calls r.cleanUp() if set 14.
• Libgit2RepoCollector.Close — defers r.cleanup() and closes the store 15.

Plugin-level Close (impl.go) is called by the framework at task teardown. It reads GIT_EXTRACTOR_KEEP_REPO from config — if unset, it calls taskData.GitRepo.Close(ctx), which triggers the cleanup callback and removes the temp dir 16.

Note: Set GIT_EXTRACTOR_KEEP_REPO=true in .env to retain the cloned directory for debugging.

Repo Collector Backends#

Two backend implementations share the RepoCollector interface:

Both implement SetCleanUp(f func()) error, Close(ctx) error, and methods for collecting tags, branches, commits, and diff lines.

Shallow-clone edge case: Both collectors skip computing commit statistics when a commit has parents that are not present in the ODB (i.e., the boundary of a shallow clone), to avoid writing inflated addition counts to the database 18 19.

Key Files#