Pipeline Monitoring#

DevLake provides two complementary mechanisms for tracking pipeline execution: REST polling (pull) and webhook notifications (push). These cover the full lifecycle from pipeline creation through completion or failure.

REST Polling (Pull Model)#

The pipeline API exposes endpoints for querying status on demand:

Pipeline statuses used in polling responses are: TASK_CREATED, TASK_RUNNING, TASK_COMPLETED, TASK_FAILED, TASK_PARTIAL, TASK_CANCELLED, TASK_RERUN, TASK_RESUME. Status is computed by ComputePipelineStatus, which aggregates individual task outcomes:

• TASK_COMPLETED — all tasks succeeded
• TASK_FAILED — at least one task failed and SkipOnFail=false
• TASK_PARTIAL — SkipOnFail=true with at least one success and one failure

Use the status query parameter on GET /pipelines to filter to a specific state (e.g., ?status=TASK_RUNNING).

Webhook Notifications (Push Model)#

DevLake fires an outbound HTTP POST to a configured endpoint on two pipeline lifecycle events:

1. Pipeline started — triggered in dequeuePipeline when a pipeline transitions from pending to running.
2. Pipeline finished — triggered at the end of runPipeline after the final status is persisted.

Both events call NotifyExternal, which resolves the notification service and invokes PipelineStatusChanged with a PipelineNotificationParam payload:

{
  "ProjectName": "projectName",
  "PipelineID": 39,
  "BeganAt": "2024-09-05T15:08:00.389+08:00",
  "FinishedAT": "2024-09-05T15:28:00.389+08:00",
  "Status": "TASK_COMPLETED",
  "CreatedAt": "2024-09-05T15:08:00.389+08:00",
  "UpdatedAt": "2024-09-05T15:28:00.785+08:00"
}

1

ProjectName is empty when the pipeline is created directly via API rather than through a blueprint 2.

Configuration#

Set in .env 3:

NOTIFICATION_ENDPOINT=https://your-receiver.example.com/hook
NOTIFICATION_SECRET=your-secret

The service is initialized at startup in pipelineServiceInit: if NOTIFICATION_ENDPOINT is non-empty, a DefaultPipelineNotificationService is created 4.

Request Format & Signature Verification#

The notification is delivered as POST {NOTIFICATION_ENDPOINT}?nouce={id}-{random}&sign={sha256} 5.

The signature is computed as 6 7:

SHA256( requestBody + NOTIFICATION_SECRET + nouce )

Verify the sign query parameter against this formula to authenticate the sender. Leave NOTIFICATION_SECRET empty to skip verification.

Notification Service Architecture#

The notification subsystem uses a two-level interface 8:

• PipelineNotificationService — interface with a single PipelineStatusChanged method.
• GetPipelineNotificationService() — returns a custom implementation if registered, then falls back to DefaultPipelineNotificationService, or nil if neither is configured.

To plug in a custom notifier (e.g., for Slack, PagerDuty), assign an implementation to customPipelineNotificationService. The built-in DefaultPipelineNotificationService persists each notification to the database before sending, recording the HTTP response code and body for auditability.

Key Source Files#