Repositories in Zerobyte#

Overview#

Repositories are storage locations where encrypted backups are securely stored. They serve as the destination for backup data and leverage Restic's deduplication and encryption features for storage efficiency and data integrity.

Repository Types#

Zerobyte supports 8 different repository backend types, each designed for different storage scenarios:

1. Local (local)#

Stores backups on local disk in a subfolder of /var/lib/zerobyte/repositories/ or any other mounted path.

Configuration:

• backend: 'local'
• path: File system path
• isExistingRepository: Boolean flag indicating whether to import an existing repository
  • When creating a new repository (false): A unique subdirectory is automatically appended to the path
  • When importing an existing repository (true): The exact path is preserved

2. S3 (s3)#

Amazon S3 and S3-compatible storage (MinIO, Wasabi, DigitalOcean Spaces, etc.)

Configuration:

• backend: 's3'
• endpoint: S3 endpoint URL
• bucket: Bucket name
• accessKeyId: Access key (encrypted)
• secretAccessKey: Secret key (encrypted)

3. R2 (r2)#

Cloudflare R2 storage

Configuration:

• backend: 'r2'
• endpoint: R2 endpoint URL
• bucket: Bucket name
• accessKeyId: Access key (encrypted)
• secretAccessKey: Secret key (encrypted)

4. GCS (gcs)#

Google Cloud Storage

Configuration:

• backend: 'gcs'
• bucket: Bucket name
• projectId: GCP project ID
• credentialsJson: Service account credentials (encrypted)

5. Azure (azure)#

Microsoft Azure Blob Storage

Configuration:

• backend: 'azure'
• container: Container name
• accountName: Storage account name
• accountKey: Account key (encrypted)
• endpointSuffix: Optional endpoint suffix

6. rclone (rclone)#

Support for 40+ cloud storage providers via rclone (Google Drive, Dropbox, OneDrive, Box, pCloud, Mega, etc.)

Configuration:

• backend: 'rclone'
• remote: rclone remote name
• path: Path within remote
• insecureTls: Optional flag to disable TLS certificate validation

7. REST (rest)#

REST server backend

Configuration:

• backend: 'rest'
• url: REST server URL
• username: Optional username (encrypted)
• password: Optional password (encrypted)
• path: Optional path

8. SFTP (sftp)#

SSH File Transfer Protocol

Configuration:

• backend: 'sftp'
• host: SSH host
• port: SSH port (default: 22)
• user: SSH username
• path: Path on remote server
• privateKey: SSH private key (encrypted)
• skipHostKeyCheck: Skip host key verification (default: true)
• knownHosts: Optional known hosts file
• allowLegacySshRsa: Enable legacy SSH RSA/SHA1 algorithms (default: false)
  • Permits RSA/SHA1 signatures for compatibility with older SFTP servers that only offer ssh-rsa host key algorithms
  • Should only be enabled for legacy servers that require it, as RSA/SHA1 signatures are weaker than modern SSH algorithms

Repository Architecture#

Database Schema#

Repositories are stored in the database with the following properties:

Core Properties:

• id: Primary UUID identifier
• shortId: Human-readable short identifier
• provisioningId: Nullable string identifying operator-managed repositories synced from a provisioning file
• name: User-friendly name
• type: Repository backend type
• config: JSON configuration with backend-specific settings
• organizationId: Multi-tenancy support

Compression:

• compressionMode: 'off', 'auto', or 'max'

Health Status:

• status: 'healthy', 'error', 'unknown', 'doctor', or 'cancelled'
• lastChecked: Timestamp of last health check
• lastError: Error message if any
• doctorResult: Diagnostic operation results

Statistics:

• stats: Stored repository statistics (compression and storage metrics)
• statsUpdatedAt: Timestamp of last statistics update

Bandwidth Limiting:

• Upload limits: uploadLimitEnabled, uploadLimitValue, uploadLimitUnit
• Download limits: downloadLimitEnabled, downloadLimitValue, downloadLimitUnit
• Units: Kbps, Mbps, or Gbps

Code Organization#

The repository functionality is organized in layers:

1. Database Schema - Data model definition
2. Configuration Schemas - Backend type definitions and schemas in @zerobyte/core package
3. Restic Core - Restic instance with dependency injection (ResticDeps interface)
4. Service Layer - Business logic
5. Controller - HTTP endpoints
6. API DTOs - Data transfer objects

@zerobyte/core Package:
The restic functionality is organized in a shared @zerobyte/core package that provides:

• @zerobyte/core/restic: Core schemas, types, and DTOs
• @zerobyte/core/restic/server: Server-side utilities (buildRepoUrl, buildEnv, cleanupTemporaryKeys, createRestic)
• @zerobyte/core/utils: Common utilities shared across the codebase
• @zerobyte/core/node: Node.js-specific utilities (logger, spawn)

The restic instance in app/server/core/restic.ts uses the ResticDeps interface to inject required dependencies (secret resolution, organization password lookup, configuration) into the restic operations, enabling better testability and modularity.

How Repositories Work#

Repository Creation Modes#

Repositories can be created through two modes:

1. Manual UI Creation: Users create and configure repositories through the Zerobyte UI
2. Operator Provisioning: Repositories are defined in a JSON configuration file and synced at startup

Provisioned/Managed Repositories#

Repositories can be provisioned from a JSON configuration file at startup by setting the PROVISIONING_PATH environment variable. This enables operator-managed infrastructure patterns where repositories are defined in configuration rather than created through the UI.

Key Characteristics:

• Provisioned repositories are marked with a non-null provisioningId field in the format provisioned:{organizationId}:{id}
• They appear in the normal repositories list with a "Managed" badge in the UI
• Secret values in the provisioning file support special protocols that are resolved at sync time:
  • env://VAR_NAME - resolves to the value of environment variable VAR_NAME
  • file://SECRET_NAME - reads from /run/secrets/SECRET_NAME (Docker secrets)
• Resolved secrets are encrypted before storage using the same encryption methods as manually created repositories
• Managed repositories appear alongside manually created repositories in all UI views and API responses
• Changes to the provisioning file are applied on the next startup/restart

Provisioning File Format:
The provisioning file is a JSON document with the following structure:

{
  "version": 1,
  "repositories": [
    {
      "id": "unique-id",
      "organizationId": "existing-org-id",
      "name": "Repository Name",
      "backend": "s3",
      "compressionMode": "auto",
      "config": {
        "backend": "s3",
        "endpoint": "https://s3.amazonaws.com",
        "bucket": "my-bucket",
        "accessKeyId": "env://AWS_ACCESS_KEY_ID",
        "secretAccessKey": "file://aws_secret_key"
      }
    }
  ]
}

Sync Behavior:

• At startup, Zerobyte reads the provisioning file and syncs all defined repositories
• For new repositories (isExistingRepository: false), restic.init() is automatically called to initialize the repository
• Existing provisioned repositories are updated if their configuration changes
• Repositories with "delete": true are removed from the database
• The sync operation is atomic - partial failures do not leave an inconsistent state

For complete examples, see the examples/provisioned-resources directory in the repository.

Repository Initialization#

New Repositories#

When creating a new repository, zerobyte:

1. Generates unique identifiers (UUID and short ID)
2. For local repositories, when isExistingRepository is false (i.e., creating a new repository), automatically appends the short ID to the specified path to create a unique subdirectory (e.g., if user specifies /my/path, it becomes /my/path/{shortId}). This prevents path conflicts when multiple repositories are created in the same directory.
3. Encrypts sensitive configuration fields
4. Inserts the repository record with status: "unknown"
5. Calls restic.init() to initialize the Restic repository
6. If appConfig.resticHostname is configured, automatically adds a custom key with that hostname using restic.keyAdd(). If this key addition fails, a warning is logged but the repository initialization still succeeds.
7. Updates status to "healthy" on success, or deletes the record on failure

The initialization is atomic - if Restic initialization fails, the database record is deleted to maintain consistency.

Initialization Timeouts#

Repository initialization operations use a dynamic timeout configuration:

• Timeout Duration: The timeout is based on the SERVER_IDLE_TIMEOUT environment variable (converted to milliseconds), which defaults to 60 seconds
• Configuration: This timeout can be adjusted through server settings to accommodate different deployment requirements (e.g., slow network connections or large repositories)
• Error Detection: Timeout errors are explicitly detected and reported with a clear error message: "Command timed out before completing"
• Consistency: This ensures that repository initialization timeout behavior is consistent with other server operations
• API Request Timeout: All API requests, including repository initialization and snapshot operations, use the server idle timeout setting via the prepareApiRequest function, which configures the Node.js response timeout to match the configured SERVER_IDLE_TIMEOUT value

Existing Repositories#

When importing an existing repository (identified by the isExistingRepository flag), zerobyte validates connectivity by listing snapshots instead of initializing a new repository. For local repositories, the specified path is preserved exactly as provided without any modifications, allowing users to import repositories from any location.

Restic Integration#

Restic Operations#

The restic utility exports the following operations:

• init: Initialize a new repository
• keyAdd: Add a custom key with a specific hostname to an existing repository
• backup: Create a backup snapshot
• restore: Restore data from a snapshot
• dump: Stream snapshot contents
• snapshots: List snapshots in the repository
• ls: List files in a snapshot
• forget: Remove snapshots
• prune: Remove unused data
• check: Verify repository integrity
• stats: Get repository statistics
• unlock: Remove stale locks
• tag: Add or remove snapshot tags
• copy: Copy snapshots between repositories

Building Repository URLs#

The buildRepoUrl() function (from @zerobyte/core/restic/server) constructs backend-specific repository URLs. For S3 and R2 backends, the endpoint is normalized by trimming leading/trailing whitespace and removing any trailing slash before building the URL. For R2, the protocol (http:// or https://) is also stripped after trimming. This ensures that extra spaces or slashes in the endpoint do not affect the resulting repository URL.

• Local: config.path
• S3: s3:endpoint/bucket (where endpoint is trimmed of whitespace and trailing slash)
• R2: s3:endpoint/bucket (where endpoint is trimmed of whitespace and trailing slash, and protocol is removed)
• GCS: gs:bucket:/
• Azure: azure:container:/
• rclone: rclone:remote:path
• REST: rest:url/path
• SFTP: sftp:user@host:path

Example:

• S3 endpoint https://s3.amazonaws.com/ and bucket my-bucket → s3:https://s3.amazonaws.com/my-bucket
• R2 endpoint https://myaccount.r2.cloudflarestorage.com/ and bucket my-bucket → s3:myaccount.r2.cloudflarestorage.com/my-bucket

Building Environment Variables#

The buildEnv() function (from @zerobyte/core/restic/server) sets up backend-specific credentials as environment variables:

S3/R2:

• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY
• AWS_REGION (R2: "auto")
• AWS_S3_FORCE_PATH_STYLE (R2: "true")

GCS:

• GOOGLE_PROJECT_ID
• GOOGLE_APPLICATION_CREDENTIALS (temporary file path)

Azure:

• AZURE_ACCOUNT_NAME
• AZURE_ACCOUNT_KEY
• AZURE_ENDPOINT_SUFFIX (optional)

REST:

• RESTIC_REST_USERNAME (optional)
• RESTIC_REST_PASSWORD (optional)

SFTP:

• SSH key written to temporary file with restricted permissions (0o600)
• SSH arguments configured via _SFTP_SSH_ARGS
• Optional legacy SSH RSA/SHA1 algorithm support: When allowLegacySshRsa is enabled, adds -o HostKeyAlgorithms=+ssh-rsa -o PubkeyAcceptedAlgorithms=+ssh-rsa to SSH arguments for compatibility with older servers

rclone:

• RCLONE_CONFIG: Path to the rclone configuration file (derived from RCLONE_CONFIG_DIR environment variable, defaulting to /root/.config/rclone/rclone.conf)
• RCLONE_NO_CHECK_CERTIFICATE: Set to "true" when insecureTls is enabled

Common:

• RESTIC_PASSWORD_FILE: Path to password file
• RESTIC_CACERT: Path to CA certificate (if provided)
• RESTIC_CACHE_DIR: Cache directory location
• _INSECURE_TLS: Set to "true" when insecureTls is enabled (applies to all backends)

All temporary credential files are cleaned up after operations complete.

Password Management#

Zerobyte uses a two-tier password system:

Organization-Level Passwords#

Each organization has a Restic password stored in encrypted metadata. When creating an organization, a random password is generated and encrypted.

Custom Repository Passwords#

Repositories can optionally have custom passwords for existing repositories.

Password Resolution#

The buildEnv() function determines which password to use:

1. If isExistingRepository and customPassword are set, use the custom password
2. Otherwise, use the organization's default password

Passwords are written to temporary files with restricted permissions (0o600) and referenced via RESTIC_PASSWORD_FILE.

Security and Encryption#

Sensitive configuration fields are encrypted before storage using cryptoUtils.sealSecret(). This includes:

• Passwords
• CA certificates
• Cloud storage access keys/credentials
• SFTP private keys
• REST server credentials

The system supports three secret storage methods:

• Direct encryption: encv1:... - encrypted in database using AES-256-GCM
• Environment variables: env://VAR_NAME - reference to process.env.VAR_NAME
• Secret files: file://name - reads from /run/secrets/name (Docker secrets)

Repository Operations#

Repositories support the following operations:

Error Handling:
All repository operations that interact with restic return structured error responses with two fields:

• message: A human-readable error summary describing the type of failure
• details (optional): Detailed diagnostic information from the underlying operation, particularly stderr output from restic commands

This structure separates high-level error summaries from detailed diagnostic information, allowing clients to display appropriate error information based on context. The details field contains specific stderr output that helps diagnose issues like connection problems, authentication failures, or file system errors.

CRUD Operations:

• Create
• Read
• Update
• Delete (note: only removes the database record, not actual data)

Snapshot Management:

• List snapshots
• Get snapshot details
• Browse files - The SnapshotTreeBrowser component provides snapshot browsing functionality with path handling that separates display and query concerns:
  • queryBasePath: The path used for API queries to fetch snapshot files (typically the common ancestor of snapshot paths)
  • displayBasePath: The path prefix used for displaying file paths in the UI (typically the volume mount path)
  • This separation allows proper rendering of ancestor folders when browsing snapshots with a nested query root - the display path can show the full hierarchy while queries are scoped to a specific subtree
• Delete snapshots: Remove one or more snapshots from the repository
  • Endpoint:
    • DELETE /api/v1/repositories/{shortId}/snapshots/{snapshotId} (single snapshot)
    • DELETE /api/v1/repositories/{shortId}/snapshots (bulk deletion with snapshotIds array in request body)
  • Returns: 202 Accepted with taskId and status: "started" to indicate deletion has started asynchronously
  • Asynchronous Execution: Snapshot deletion is an asynchronous operation that returns immediately with a task ID for tracking. The actual deletion happens in the background as a task.
  • Task-Based Event Streaming: Snapshot deletion is tracked through the task-based event streaming system. The client subscribes to task events via /api/v1/tasks/events with filters for the repository resource (kind=deleteSnapshots, resourceType=repository, resourceId={repositoryId}). Tasks are tracked using task:changed and tasks:snapshot events. The useTaskEvents hook manages real-time task updates for the repository's delete operations.
  • Statistics Refresh: Repository statistics are automatically refreshed in the background after successful deletion completes
• Tag snapshots: Add, remove, or set tags on one or more snapshots
  • Endpoint: POST /api/v1/repositories/{shortId}/snapshots/tag
  • Parameters:
    • snapshotIds: Array of snapshot IDs to tag (minimum 1)
    • add: Array of tags to add (optional)
    • remove: Array of tags to remove (optional)
    • set: Array of tags to set (replaces all existing tags, optional)
  • Returns: 202 Accepted with taskId and status: "started" to indicate tagging has started asynchronously
  • Asynchronous Execution: Snapshot tagging is an asynchronous operation that returns immediately with a task ID for tracking. The actual tagging happens in the background as a task.
  • Task-Based Event Streaming: Snapshot tagging is tracked through the task-based event streaming system. The client subscribes to task events via /api/v1/tasks/events with filters for the repository resource (kind=tagSnapshots, resourceType=repository, resourceId={repositoryId}). The useTaskEvents hook manages real-time task updates for the repository's tag operations.
  • UI Feedback: The UI shows a loading spinner for snapshots being tagged, and snapshots being processed cannot be selected for other operations
• Download snapshots

Restore Operations:

• Restore snapshot data
  • Endpoint: POST /api/v1/repositories/{shortId}/snapshots/{snapshotId}/restore
  • Parameters:
    • snapshotId: The snapshot to restore from
    • include: Array of paths to include (optional)
    • selectedItemKind: "file" or "dir" - specifies the type when restoring a single item, helping the system determine whether the selected path is a file or directory (optional)
    • exclude: Array of paths to exclude (optional)
    • excludeXattr: Array of extended attributes to exclude (optional)
    • targetPath: Custom location for restore
    • targetAgentId: Agent ID that should execute the restore operation (optional, enables restoring to remote agents)
    • overwrite: Overwrite mode for existing files ('always', 'if-changed', 'if-newer', or 'never')
  • Returns: 202 Accepted with restoreId and status: "started" to indicate the restore operation has started asynchronously
  • Asynchronous Execution: The restore operation returns immediately with a restoreId for tracking progress. The operation uses task-based orchestration and emits events (restore:started, restore:progress, restore:completed) for real-time monitoring.
  • Concurrency Control: Only one restore can be active per snapshot at a time. Attempting to start a restore when one is already running returns 409 Conflict.
  • Remote Agent Support: The targetAgentId parameter allows specifying which agent should execute the restore. Local repositories must restore on the agent that can access the repository path. The target agent must belong to the same organization.
  • Cancellation: Restore operations can be cancelled while running via the generic task cancellation endpoint: POST /api/v1/tasks/{taskId}/cancel. Returns {status: "cancelling"} (202 Accepted). The UI provides a "Cancel restore" button during active restore operations. When cancelled, the restore task status changes to "cancelled", and users receive a dialog notification: "Restore cancelled" with a message like "Restore of snapshot {snapshotId} was cancelled."
  • Single-file restore behavior: When restoring a single file to a custom location with selectedItemKind: "file", the system uses the file's parent directory as the common ancestor for proper path resolution, ensuring the file is placed correctly at the target path
  • The RestoreForm component accepts both queryBasePath and displayBasePath parameters for path handling:
    • queryBasePath: The base path for API queries (scoped to snapshot contents)
    • displayBasePath: The path prefix for displaying file paths in the UI
  • Cross-Platform Path Compatibility: Snapshots created with non-POSIX file paths (e.g., Windows paths like C:\Users\...) cannot be restored to their original location. When such paths are detected:
    • The "Original location" restore option is automatically disabled in the UI
    • A warning is displayed: "Source paths do not match - This snapshot was created from source paths that do not match this Zerobyte server or the current linked volume. Restoring to the original location is unavailable. Restore it to a custom location, or download it instead."
    • These snapshots must be restored to a custom target location (the targetPath parameter is required)
    • Attempting to restore non-POSIX snapshots to their original location via API results in a BadRequestError: "Original location restore is unavailable for this snapshot. Restore it to a custom location instead."
  • Protected path restrictions: To prevent accidental overwriting of critical system files or application data, restore operations are blocked from targeting certain protected paths. If a restore is attempted to any of these locations, the operation will fail with a BadRequestError: "Restore target path is not allowed. Restoring to this path could overwrite critical system files or application data." The following paths and their subdirectories are protected from restore operations:
    • Repository base directory (/var/lib/zerobyte/repositories)
    • Restic cache directory (/var/lib/zerobyte/cache)
    • SSH keys directory (/var/lib/zerobyte/ssh)
    • Rclone config directory (/root/.config/rclone)
    • Application directory (/app)
    • Database directory
    • Restic password file directory
    • System temp directory
    • Provisioning directory (if configured)

Repository Statistics:

• Get repository statistics: Retrieve stored compression and storage statistics

  • Endpoint: GET /api/v1/repositories/{shortId}/stats
  • Returns stored statistics from the database (stats and statsUpdatedAt columns)
  • When a repository has no statistics yet (new repository or after configuration changes), returns empty stats (all zeros) with a message indicating stats will be populated after the first backup or manual refresh
  • Statistics are automatically refreshed in the background after:
    • Successful backup completion
    • Snapshot deletion (both single and bulk operations)
  • Statistics can be manually refreshed using the refresh endpoint (see below)
  • Returns ResticStatsDto with the following fields:
    • total_size: Total size of stored data in bytes
    • total_uncompressed_size: Total uncompressed size in bytes
    • compression_ratio: Compression ratio (e.g., 2.5x)
    • compression_progress: Percentage of snapshots that are compressed (0-1)
    • compression_space_saving: Space saving percentage as decimal (0-1)
    • snapshots_count: Total number of snapshots in the repository

• Refresh repository statistics: Manually update stored statistics

  • Endpoint: POST /api/v1/repositories/{shortId}/stats/refresh
  • Runs restic stats command and updates the database with fresh statistics
  • Acquires a shared repository lock to safely read stats while allowing concurrent operations
  • Returns the updated ResticStatsDto
  • Background refresh errors are logged but do not block the triggering operation

Download Snapshots#

The dumpSnapshot() method provides snapshot download functionality through the /api/v1/repositories/{shortId}/snapshots/{snapshotId}/dump endpoint:

Query Parameters:

• path (optional): Specific path within the snapshot to download
• kind (optional): Type of path - "file" or "dir" (required when path is specified)

Behavior:

• Without parameters: Downloads entire snapshot as a tar archive
• With path pointing to a directory (kind: "dir"): Downloads directory contents as tar archive
• With path pointing to a file (kind: "file"): Downloads raw file stream with application/octet-stream content type

Implementation Details:

• Uses the prepareSnapshotDump() helper to process paths and determine the appropriate snapshot reference
• Acquires a shared repository lock with key dump:${snapshotId} to prevent conflicts during the operation
• Emits a dump:started server event containing organizationId, repositoryId, snapshotId, path, and filename
• Returns a stream with completion promise, filename, and appropriate content type
• Sets proper Content-Type and Content-Disposition headers for file downloads
• Defaults to filename pattern snapshot-<snapshotId>.tar or uses the actual filename for individual files
• Uses a custom ReadableStream implementation that wraps the source dump stream with pull/cancel handlers to ensure downloads continue even after HTTP request signals abort
• The cancel handler explicitly calls dumpStream.abort() only when the stream is truly cancelled, not when the request signal aborts

Health Checks:

• Check repository health
• Run doctor operations (unlock, check, repair index)

Lock Management:

• Unlock repository

Health Monitoring#

The checkHealth() function runs Restic's check command to verify repository integrity and updates the status accordingly.

Doctor Operations#

The doctor operation performs automated repository maintenance through a task-based architecture. It executes three steps sequentially:

1. Unlock: Removes stale locks
2. Check: Verifies repository integrity
3. Repair Index: Rebuilds index if suggested by check

Task-Based Architecture:
Doctor operations run as part of the task stream system. When a doctor operation is initiated, a task with kind: "doctor" is created and managed by the task lifecycle system.

Implementation:

• Server-side: app/server/modules/repositories/commands/doctor-command.ts
• Client-side hooks: app/client/modules/repositories/doctor-tasks.ts

API Endpoints:

• Start doctor: POST /api/v1/repositories/{shortId}/doctor
  • Returns: {taskId: string, status: "started"}
• Cancel doctor: POST /api/v1/tasks/{taskId}/cancel
  • Returns: {status: "cancelling"} (202 Accepted)
  • Error: 409 Conflict - "Task is not cancellable or is no longer running"

Task Schema:

• Input: {kind: "doctor", repositoryId: string}
• Result: {kind: "doctor", repositoryStatus: "healthy" | "error" | "cancelled", lastChecked: number, lastError: string | null, doctorResult: DoctorResult}

Server-Sent Events:
Doctor operations emit generic task lifecycle events that clients can filter:

• task:started event: {organizationId, taskId, kind: "doctor", resourceType: "repository", resourceId, status}
• task:finished event: {organizationId, taskId, kind: "doctor", resourceType: "repository", resourceId, status}

Clients listen for these events filtered by kind="doctor" and resourceType="repository" to track doctor progress in real-time.

Cancellation:
Doctor operations are cancellable using the generic task cancellation endpoint: POST /api/v1/tasks/{taskId}/cancel. When cancelled, the task's abort signal is triggered, allowing in-progress restic operations to be interrupted gracefully. The task result includes a repositoryStatus field indicating whether the operation was cancelled.

Storage:
Results are stored in the doctorResult field for troubleshooting.

Concurrency Control#

Repository operations use a database-backed mutex system to prevent conflicts:

Lock Types:

• Shared locks: Multiple read operations (snapshots, ls, restore, dump) can run concurrently
• Exclusive locks: Write operations (backup, forget, check, doctor) require exclusive access
• Operations wait in a queue (FIFO order) when locks are held

Database Persistence:
Locks are stored in two database tables:

• repository_locks: Active locks with fields including id, repositoryId, type, operation, ownerId, acquiredAt, expiresAt, and heartbeatAt
• repository_lock_waiters: Queued operations waiting to acquire locks with fields including requestedAt, expiresAt, and heartbeatAt

Lock Leasing and Automatic Expiration:

• Locks use a 30-second lease (LOCK_LEASE_MS = 30,000ms)
• Active locks send heartbeats every 5 seconds (LOCK_HEARTBEAT_MS = 5,000ms)
• Locks expire if not renewed, providing automatic cleanup of stale locks from crashed processes
• Locks persist across application restarts, preventing lost locks

Queue Management:

• Waiters poll the database every 250ms (LOCK_POLL_MS) to check for lock availability
• Expired locks and waiters are cleaned up every 5 seconds during polling (LOCK_POLL_CLEANUP_MS)
• Each lock/waiter is associated with an ownerId (unique per process instance) to ensure only the owning process can release or update its locks

This database-backed implementation ensures data consistency and prevents Restic repository corruption from concurrent modifications while providing robustness through lock persistence and automatic expiration.

Integration with Backup System#

Repositories are referenced by backup schedules and can serve as mirror destinations for copying snapshots to secondary storage locations.

Backup Execution Flow#

During backup execution, the system:

1. Acquires a shared lock on the repository
2. Calls restic.backup() with the repository configuration (including any custom restic parameters)
3. Emits progress events
4. Releases the lock when complete

When a backup job executes, any custom restic parameters configured in the schedule are passed directly to the restic backup command, in addition to the standard backup configuration.

Automatic vs. Manual Execution:

• Schedules with a cronExpression are automatically executed by the cron job according to their schedule
• Manual-only schedules (with empty or null cronExpression) are skipped by automatic execution and must be triggered manually by users
• After each backup execution (whether automatic or manual), the nextBackupAt field is recalculated based on the cronExpression. For manual-only schedules, this field remains null

Backup Schedule Configuration:
Backup schedules support two separate fields for specifying which files to include:

• includePaths: An array of exact file or directory paths (raw, absolute paths). These are typically selected through the file browser in the UI and are passed to restic using the --files-from-raw option with null-separated values.
• includePatterns: An array of glob patterns for matching files/directories to include. These patterns are passed to restic using the --files-from option with newline-separated values.

Both fields are optional and can be used independently or together. When neither is specified, the backup includes the entire source directory.

Manual-Only Schedules:
Backup schedules can be created without a cron expression for manual-only backups:

• When cronExpression is empty or null, the schedule displays "Manual only" in the UI
• Manual-only schedules have enabled=false and nextBackupAt=null since they will not run automatically
• Users must manually trigger backups for manual-only schedules through the UI

Validation:

• If a schedule has enabled=true, it must have a valid cronExpression
• Empty cronExpression is only allowed when enabled=false (manual-only mode)
• When creating or updating a schedule, the system validates this constraint and returns an error if violated

Custom Restic Parameters#

Backup schedules support custom restic parameters that allow advanced users to customize restic behavior for specific backup scenarios. This is an advanced feature for users familiar with restic command-line options.

Configuration:

• Stored in the customResticParams field in the database as an array of strings
• Configured through an "Advanced" section in the create/edit schedule form
• Users can specify arbitrary restic flags (e.g., --ignore-inode --ignore-ctime for performance optimization, --iexclude-larger-than 500M, --no-scan, --read-concurrency 8)

Implementation:

• Parameters are stored as an array in the backup schedule configuration
• During backup execution, parameters are passed to the restic.backup() command
• Each parameter string is tokenized by splitting on whitespace, and tokens are appended directly to the restic command arguments
• This provides flexibility to pass any restic-supported flags without requiring code changes

Usage Considerations:
Users should be cautious with custom parameters as they can affect backup behavior and compatibility. These parameters are applied in addition to the standard backup configuration and may override or conflict with default settings.

UI:
The create/edit schedule form includes an "Advanced" section with a textarea where users can enter custom restic parameters, one flag per line. The form provides examples and descriptions to guide users in specifying valid flags.

The frequency dropdown in the create/edit schedule form includes the following options:

• Manual only
• Hourly
• Daily
• Weekly
• Monthly
• Custom (cron expression)

Selecting "Manual only" creates a schedule without automatic execution.

UI Components#

Repositories Table#

The repositories table provides a browsable interface for managing repositories with enhanced visual design.

Location: app/client/modules/repositories/routes/repositories.tsx

Visual Design:

• Table Row Styling: Standardized row height of h-12 for consistent appearance across all pages
• Hover Effects: Subtle transitions with hover:bg-white/2 background and hover:border-white/10 border effects, replacing the previous hover:bg-accent/50
• Typography: Repository names display in monospace font (font-mono) with text-strong-accent color for better readability and technical aesthetics
• Repository Types: Backend types shown with text-muted-foreground color in monospace styling
• Status Display: Repository statuses are displayed using a StatusDot component that maps repository status values to visual variants. The component maps healthy status to success variant, error status to error variant, and other statuses (including unknown) to warning variant. Status labels are capitalized (e.g., "Healthy", "Error", "Unknown").
• Managed Badge: Provisioned repositories display a "Managed" badge next to their name, indicating they are controlled by the provisioning configuration file
• Footer Formatting: Repository count displayed with font-mono styling, with the count number highlighted using text-strong-accent font-bold for emphasis

Interactive States:
The dashboard redesign improved interactive states and visual consistency throughout the repositories interface, providing better feedback for user interactions with smooth transitions and refined hover effects.

Repository Details Page Layout#

The repository details page has been restructured to provide a cleaner, more organized interface.

Location: app/client/modules/repositories/routes/repository-details.tsx

Page Structure:

• Main header card: Displays repository name, status, type, and timestamps (created date, last checked)
• Actions: Repository management actions (Delete, Edit, Unlock, Run Doctor) are located in the page header via an "Actions" dropdown menu
• Last Error card: Shown below the header when the repository has an error status
• Tabs: Configuration and Snapshots tabs for detailed repository information

CompressionStatsChart#

The CompressionStatsChart component visualizes repository compression statistics in a card format.

Location: app/client/modules/repositories/components/compression-stats-chart.tsx

Displayed on: Repository info tab (rendered within a grid layout alongside the repository overview card)

Purpose: Provides visual feedback on repository storage efficiency and compression effectiveness

Visualization:

• Horizontal bar chart showing storage utilization:
  • "On disk" section (displayed in accent color) - represents the percentage of space used by stored size
  • "Freed by compression" label (shown only when stored percentage is less than 80%)
  • Below the bar: stored size on the left, and amount freed by compression on the right
• Chart header: Displays uncompressed size and snapshot count as context text

Metrics displayed:
The component displays three inline metrics below the chart:

• Ratio: Compression ratio (e.g., "2.45x")
• Snapshots: Number of snapshots
• Compressed: Compression progress percentage (displayed with 0 decimal places)

Data Loading:

• Statistics are fetched from the /api/v1/repositories/{shortId}/stats endpoint using React Query (getRepositoryStatsOptions)
• Returns stored statistics from the database, not computed on-demand
• Empty statistics are displayed when a repository has no stats yet (before first backup or after configuration changes)
• The component displays a message indicating stats will be populated after the first backup or manual refresh

Manual Refresh:

• Includes a refresh button (RefreshCw icon) that allows users to manually update statistics
• Clicking the refresh button calls the POST /api/v1/repositories/{shortId}/stats/refresh endpoint
• Shows a loading spinner while refreshing
• Displays success/error toasts after refresh completes
• Users should refresh statistics after operations that change repository contents to see updated metrics

Implementation:

• Handles loading states while fetching statistics
• Displays error messages if statistics cannot be retrieved
• Shows empty state with guidance when no statistics are available yet
• Formats byte sizes and percentages for readability using the ByteSize component
• Uses safe number parsing to handle edge cases and prevent display of invalid values

LocalRepositoryForm#

The LocalRepositoryForm component provides a form interface for creating or importing local repositories.

Location: app/client/modules/repositories/components/repository-forms/local-repository-form.tsx

Context-Aware Path Handling:
The form displays different descriptions and UI elements based on the isExistingRepository flag:

Creating New Repositories:

• Description: "A unique subdirectory will be created inside this directory to store the repository."
• Path display includes a visual preview with /{unique-id} suffix to indicate automatic subdirectory generation
• Example: User enters /my/path, displayed as /my/path/{unique-id}

Importing Existing Repositories:

• Description: "The exact path to your existing repository."
• Path display shows only the specified path without modification
• Example: User enters /my/path/existing-repo, displayed as-is

This visual feedback helps users understand the path handling behavior before creating or importing a repository.