Repositories in Zerobyte#
Overview#
Repositories are storage locations where encrypted backups are securely stored. They serve as the destination for backup data and leverage Restic's deduplication and encryption features for storage efficiency and data integrity.
Repository Types#
Zerobyte supports 8 different repository backend types, each designed for different storage scenarios:
1. Local (local)#
Stores backups on local disk in a subfolder of /var/lib/zerobyte/repositories/ or any other mounted path.
Configuration:
backend:'local'path: File system pathisExistingRepository: Boolean flag indicating whether to import an existing repository- When creating a new repository (
false): A unique subdirectory is automatically appended to the path - When importing an existing repository (
true): The exact path is preserved
- When creating a new repository (
2. S3 (s3)#
Amazon S3 and S3-compatible storage (MinIO, Wasabi, DigitalOcean Spaces, etc.)
Configuration:
backend:'s3'endpoint: S3 endpoint URLbucket: Bucket nameaccessKeyId: Access key (encrypted)secretAccessKey: Secret key (encrypted)
3. R2 (r2)#
Configuration:
backend:'r2'endpoint: R2 endpoint URLbucket: Bucket nameaccessKeyId: Access key (encrypted)secretAccessKey: Secret key (encrypted)
4. GCS (gcs)#
Configuration:
backend:'gcs'bucket: Bucket nameprojectId: GCP project IDcredentialsJson: Service account credentials (encrypted)
5. Azure (azure)#
Configuration:
backend:'azure'container: Container nameaccountName: Storage account nameaccountKey: Account key (encrypted)endpointSuffix: Optional endpoint suffix
6. rclone (rclone)#
Support for 40+ cloud storage providers via rclone (Google Drive, Dropbox, OneDrive, Box, pCloud, Mega, etc.)
Configuration:
backend:'rclone'remote: rclone remote namepath: Path within remoteinsecureTls: Optional flag to disable TLS certificate validation
7. REST (rest)#
Configuration:
backend:'rest'url: REST server URLusername: Optional username (encrypted)password: Optional password (encrypted)path: Optional path
8. SFTP (sftp)#
Configuration:
backend:'sftp'host: SSH hostport: SSH port (default: 22)user: SSH usernamepath: Path on remote serverprivateKey: SSH private key (encrypted)skipHostKeyCheck: Skip host key verification (default: true)knownHosts: Optional known hosts file
Repository Architecture#
Database Schema#
Repositories are stored in the database with the following properties:
Core Properties:
id: Primary UUID identifiershortId: Human-readable short identifierprovisioningId: Nullable string identifying operator-managed repositories synced from a provisioning filename: User-friendly nametype: Repository backend typeconfig: JSON configuration with backend-specific settingsorganizationId: Multi-tenancy support
Compression:
compressionMode:'off','auto', or'max'
Health Status:
status:'healthy','error','unknown','doctor', or'cancelled'lastChecked: Timestamp of last health checklastError: Error message if anydoctorResult: Diagnostic operation results
Statistics:
stats: Stored repository statistics (compression and storage metrics)statsUpdatedAt: Timestamp of last statistics update
Bandwidth Limiting:
- Upload limits:
uploadLimitEnabled,uploadLimitValue,uploadLimitUnit - Download limits:
downloadLimitEnabled,downloadLimitValue,downloadLimitUnit - Units: Kbps, Mbps, or Gbps
Code Organization#
The repository functionality is organized in layers:
- Database Schema - Data model definition
- Configuration Schemas - Backend type definitions and schemas in
@zerobyte/corepackage - Restic Core - Restic instance with dependency injection (
ResticDepsinterface) - Service Layer - Business logic
- Controller - HTTP endpoints
- API DTOs - Data transfer objects
@zerobyte/core Package:
The restic functionality is organized in a shared @zerobyte/core package that provides:
@zerobyte/core/restic: Core schemas, types, and DTOs@zerobyte/core/restic/server: Server-side utilities (buildRepoUrl,buildEnv,cleanupTemporaryKeys,createRestic)@zerobyte/core/utils: Common utilities shared across the codebase@zerobyte/core/node: Node.js-specific utilities (logger, spawn)
The restic instance in app/server/core/restic.ts uses the ResticDeps interface to inject required dependencies (secret resolution, organization password lookup, configuration) into the restic operations, enabling better testability and modularity.
How Repositories Work#
Repository Creation Modes#
Repositories can be created through two modes:
- Manual UI Creation: Users create and configure repositories through the Zerobyte UI
- Operator Provisioning: Repositories are defined in a JSON configuration file and synced at startup
Provisioned/Managed Repositories#
Repositories can be provisioned from a JSON configuration file at startup by setting the PROVISIONING_PATH environment variable. This enables operator-managed infrastructure patterns where repositories are defined in configuration rather than created through the UI.
Key Characteristics:
- Provisioned repositories are marked with a non-null
provisioningIdfield in the formatprovisioned:{organizationId}:{id} - They appear in the normal repositories list with a "Managed" badge in the UI
- Secret values in the provisioning file support special protocols that are resolved at sync time:
env://VAR_NAME- resolves to the value of environment variableVAR_NAMEfile://SECRET_NAME- reads from/run/secrets/SECRET_NAME(Docker secrets)
- Resolved secrets are encrypted before storage using the same encryption methods as manually created repositories
- Managed repositories appear alongside manually created repositories in all UI views and API responses
- Changes to the provisioning file are applied on the next startup/restart
Provisioning File Format:
The provisioning file is a JSON document with the following structure:
{
"version": 1,
"repositories": [
{
"id": "unique-id",
"organizationId": "existing-org-id",
"name": "Repository Name",
"backend": "s3",
"compressionMode": "auto",
"config": {
"backend": "s3",
"endpoint": "https://s3.amazonaws.com",
"bucket": "my-bucket",
"accessKeyId": "env://AWS_ACCESS_KEY_ID",
"secretAccessKey": "file://aws_secret_key"
}
}
]
}
Sync Behavior:
- At startup, Zerobyte reads the provisioning file and syncs all defined repositories
- For new repositories (
isExistingRepository: false),restic.init()is automatically called to initialize the repository - Existing provisioned repositories are updated if their configuration changes
- Repositories with
"delete": trueare removed from the database - The sync operation is atomic - partial failures do not leave an inconsistent state
For complete examples, see the examples/provisioned-resources directory in the repository.
Repository Initialization#
New Repositories#
When creating a new repository, zerobyte:
- Generates unique identifiers (UUID and short ID)
- For local repositories, when
isExistingRepositoryis false (i.e., creating a new repository), automatically appends the short ID to the specified path to create a unique subdirectory (e.g., if user specifies/my/path, it becomes/my/path/{shortId}). This prevents path conflicts when multiple repositories are created in the same directory. - Encrypts sensitive configuration fields
- Inserts the repository record with
status: "unknown" - Calls
restic.init()to initialize the Restic repository - If
appConfig.resticHostnameis configured, automatically adds a custom key with that hostname usingrestic.keyAdd(). If this key addition fails, a warning is logged but the repository initialization still succeeds. - Updates status to
"healthy"on success, or deletes the record on failure
The initialization is atomic - if Restic initialization fails, the database record is deleted to maintain consistency.
Initialization Timeouts#
Repository initialization operations use a dynamic timeout configuration:
- Timeout Duration: The timeout is based on the
SERVER_IDLE_TIMEOUTenvironment variable (converted to milliseconds), which defaults to 60 seconds - Configuration: This timeout can be adjusted through server settings to accommodate different deployment requirements (e.g., slow network connections or large repositories)
- Error Detection: Timeout errors are explicitly detected and reported with a clear error message: "Command timed out before completing"
- Consistency: This ensures that repository initialization timeout behavior is consistent with other server operations
- API Request Timeout: All API requests, including repository initialization and snapshot operations, use the server idle timeout setting via the
prepareApiRequestfunction, which configures the Node.js response timeout to match the configuredSERVER_IDLE_TIMEOUTvalue
Existing Repositories#
When importing an existing repository (identified by the isExistingRepository flag), zerobyte validates connectivity by listing snapshots instead of initializing a new repository. For local repositories, the specified path is preserved exactly as provided without any modifications, allowing users to import repositories from any location.
Restic Integration#
Restic Operations#
The restic utility exports the following operations:
init: Initialize a new repositorykeyAdd: Add a custom key with a specific hostname to an existing repositorybackup: Create a backup snapshotrestore: Restore data from a snapshotdump: Stream snapshot contentssnapshots: List snapshots in the repositoryls: List files in a snapshotforget: Remove snapshotsprune: Remove unused datacheck: Verify repository integritystats: Get repository statisticsunlock: Remove stale lockstag: Add or remove snapshot tagscopy: Copy snapshots between repositories
Building Repository URLs#
The buildRepoUrl() function (from @zerobyte/core/restic/server) constructs backend-specific repository URLs. For S3 and R2 backends, the endpoint is normalized by trimming leading/trailing whitespace and removing any trailing slash before building the URL. For R2, the protocol (http:// or https://) is also stripped after trimming. This ensures that extra spaces or slashes in the endpoint do not affect the resulting repository URL.
- Local:
config.path - S3:
s3:endpoint/bucket(whereendpointis trimmed of whitespace and trailing slash) - R2:
s3:endpoint/bucket(whereendpointis trimmed of whitespace and trailing slash, and protocol is removed) - GCS:
gs:bucket:/ - Azure:
azure:container:/ - rclone:
rclone:remote:path - REST:
rest:url/path - SFTP:
sftp:user@host:path
Example:
- S3 endpoint
https://s3.amazonaws.com/and bucketmy-bucket→s3:https://s3.amazonaws.com/my-bucket - R2 endpoint
https://myaccount.r2.cloudflarestorage.com/and bucketmy-bucket→s3:myaccount.r2.cloudflarestorage.com/my-bucket
Building Environment Variables#
The buildEnv() function (from @zerobyte/core/restic/server) sets up backend-specific credentials as environment variables:
S3/R2:
AWS_ACCESS_KEY_IDAWS_SECRET_ACCESS_KEYAWS_REGION(R2: "auto")AWS_S3_FORCE_PATH_STYLE(R2: "true")
GCS:
GOOGLE_PROJECT_IDGOOGLE_APPLICATION_CREDENTIALS(temporary file path)
Azure:
AZURE_ACCOUNT_NAMEAZURE_ACCOUNT_KEYAZURE_ENDPOINT_SUFFIX(optional)
REST:
RESTIC_REST_USERNAME(optional)RESTIC_REST_PASSWORD(optional)
SFTP:
- SSH key written to temporary file with restricted permissions (0o600)
- SSH arguments configured via
_SFTP_SSH_ARGS
rclone:
RCLONE_NO_CHECK_CERTIFICATE: Set to"true"wheninsecureTlsis enabled
Common:
RESTIC_PASSWORD_FILE: Path to password fileRESTIC_CACERT: Path to CA certificate (if provided)RESTIC_CACHE_DIR: Cache directory location_INSECURE_TLS: Set to"true"wheninsecureTlsis enabled (applies to all backends)
All temporary credential files are cleaned up after operations complete.
Password Management#
Zerobyte uses a two-tier password system:
Organization-Level Passwords#
Each organization has a Restic password stored in encrypted metadata. When creating an organization, a random password is generated and encrypted.
Custom Repository Passwords#
Repositories can optionally have custom passwords for existing repositories.
Password Resolution#
The buildEnv() function determines which password to use:
- If
isExistingRepositoryandcustomPasswordare set, use the custom password - Otherwise, use the organization's default password
Passwords are written to temporary files with restricted permissions (0o600) and referenced via RESTIC_PASSWORD_FILE.
Security and Encryption#
Sensitive configuration fields are encrypted before storage using cryptoUtils.sealSecret(). This includes:
- Passwords
- CA certificates
- Cloud storage access keys/credentials
- SFTP private keys
- REST server credentials
The system supports three secret storage methods:
- Direct encryption:
encv1:...- encrypted in database using AES-256-GCM - Environment variables:
env://VAR_NAME- reference toprocess.env.VAR_NAME - Secret files:
file://name- reads from/run/secrets/name(Docker secrets)
Repository Operations#
Repositories support the following operations:
Error Handling:
All repository operations that interact with restic return structured error responses with two fields:
message: A human-readable error summary describing the type of failuredetails(optional): Detailed diagnostic information from the underlying operation, particularly stderr output from restic commands
This structure separates high-level error summaries from detailed diagnostic information, allowing clients to display appropriate error information based on context. The details field contains specific stderr output that helps diagnose issues like connection problems, authentication failures, or file system errors.
CRUD Operations:
Snapshot Management:
- List snapshots
- Get snapshot details
- Browse files - The
SnapshotTreeBrowsercomponent provides snapshot browsing functionality with path handling that separates display and query concerns:queryBasePath: The path used for API queries to fetch snapshot files (typically the common ancestor of snapshot paths)displayBasePath: The path prefix used for displaying file paths in the UI (typically the volume mount path)- This separation allows proper rendering of ancestor folders when browsing snapshots with a nested query root - the display path can show the full hierarchy while queries are scoped to a specific subtree
- Delete snapshots
- Tag snapshots
- Download snapshots
Restore Operations:
- Restore snapshot data
- Endpoint:
POST /api/v1/repositories/{shortId}/snapshots/{snapshotId}/restore - Parameters:
snapshotId: The snapshot to restore frominclude: Array of paths to include (optional)selectedItemKind:"file"or"dir"- specifies the type when restoring a single item (optional)exclude: Array of paths to exclude (optional)excludeXattr: Array of extended attributes to exclude (optional)targetPath: Custom location for restoreoverwrite: Overwrite mode for existing files ('always','if-changed','if-newer', or'never')
- Single-file restore behavior: When restoring a single file to a custom location with
selectedItemKind: "file", the system uses the file's parent directory as the common ancestor for proper path resolution, ensuring the file is placed correctly at the target path - The
RestoreFormcomponent accepts bothqueryBasePathanddisplayBasePathparameters for path handling:queryBasePath: The base path for API queries (scoped to snapshot contents)displayBasePath: The path prefix for displaying file paths in the UI
- Protected path restrictions: To prevent accidental overwriting of critical system files or application data, restore operations are blocked from targeting certain protected paths. If a restore is attempted to any of these locations, the operation will fail with a
BadRequestError: "Restore target path is not allowed. Restoring to this path could overwrite critical system files or application data."The following paths and their subdirectories are protected from restore operations:- Repository base directory (
/var/lib/zerobyte/repositories) - Restic cache directory (
/var/lib/zerobyte/cache) - SSH keys directory (
/var/lib/zerobyte/ssh) - Rclone config directory (
/root/.config/rclone) - Application directory (
/app) - Database directory
- Restic password file directory
- System temp directory
- Provisioning directory (if configured)
- Repository base directory (
- Endpoint:
Repository Statistics:
-
Get repository statistics: Retrieve stored compression and storage statistics
- Endpoint:
GET /api/v1/repositories/{shortId}/stats - Returns stored statistics from the database (
statsandstatsUpdatedAtcolumns) - When a repository has no statistics yet (new repository or after configuration changes), returns empty stats (all zeros) with a message indicating stats will be populated after the first backup or manual refresh
- Statistics are automatically refreshed in the background after:
- Successful backup completion
- Snapshot deletion (both single and bulk operations)
- Statistics can be manually refreshed using the refresh endpoint (see below)
- Returns
ResticStatsDtowith the following fields:total_size: Total size of stored data in bytestotal_uncompressed_size: Total uncompressed size in bytescompression_ratio: Compression ratio (e.g., 2.5x)compression_progress: Percentage of snapshots that are compressed (0-1)compression_space_saving: Space saving percentage as decimal (0-1)snapshots_count: Total number of snapshots in the repository
- Endpoint:
-
Refresh repository statistics: Manually update stored statistics
- Endpoint:
POST /api/v1/repositories/{shortId}/stats/refresh - Runs
restic statscommand and updates the database with fresh statistics - Acquires a shared repository lock to safely read stats while allowing concurrent operations
- Returns the updated
ResticStatsDto - Background refresh errors are logged but do not block the triggering operation
- Endpoint:
Download Snapshots#
The dumpSnapshot() method provides snapshot download functionality through the /api/v1/repositories/{shortId}/snapshots/{snapshotId}/dump endpoint:
Query Parameters:
path(optional): Specific path within the snapshot to downloadkind(optional): Type of path -"file"or"dir"(required whenpathis specified)
Behavior:
- Without parameters: Downloads entire snapshot as a tar archive
- With path pointing to a directory (
kind: "dir"): Downloads directory contents as tar archive - With path pointing to a file (
kind: "file"): Downloads raw file stream withapplication/octet-streamcontent type
Implementation Details:
- Uses the
prepareSnapshotDump()helper to process paths and determine the appropriate snapshot reference - Acquires a shared repository lock with key
dump:${snapshotId}to prevent conflicts during the operation - Emits a
dump:startedserver event containingorganizationId,repositoryId,snapshotId,path, andfilename - Returns a stream with completion promise, filename, and appropriate content type
- Sets proper
Content-TypeandContent-Dispositionheaders for file downloads - Defaults to filename pattern
snapshot-<snapshotId>.taror uses the actual filename for individual files - Uses a custom ReadableStream implementation that wraps the source dump stream with pull/cancel handlers to ensure downloads continue even after HTTP request signals abort
- The cancel handler explicitly calls
dumpStream.abort()only when the stream is truly cancelled, not when the request signal aborts
Health Checks:
- Check repository health
- Run doctor operations (unlock, check, repair index)
Lock Management:
Health Monitoring#
The checkHealth() function runs Restic's check command to verify repository integrity and updates the status accordingly.
Doctor Operations#
The "Doctor" operation performs automated maintenance:
- Unlock: Removes stale locks
- Check: Verifies repository integrity
- Repair Index: Rebuilds index if suggested by check
Results are stored in the doctorResult field for troubleshooting.
Concurrency Control#
Repository operations use a sophisticated mutex system to prevent conflicts:
- Shared locks: Multiple read operations (snapshots, ls, restore, dump) can run concurrently
- Exclusive locks: Write operations (backup, forget, check, doctor) require exclusive access
- Operations wait in a queue when locks are held
This ensures data consistency and prevents Restic repository corruption from concurrent modifications.
Integration with Backup System#
Repositories are referenced by backup schedules and can serve as mirror destinations for copying snapshots to secondary storage locations.
Backup Execution Flow#
During backup execution, the system:
- Acquires a shared lock on the repository
- Calls
restic.backup()with the repository configuration (including any custom restic parameters) - Emits progress events
- Releases the lock when complete
When a backup job executes, any custom restic parameters configured in the schedule are passed directly to the restic backup command, in addition to the standard backup configuration.
Automatic vs. Manual Execution:
- Schedules with a
cronExpressionare automatically executed by the cron job according to their schedule - Manual-only schedules (with empty or null
cronExpression) are skipped by automatic execution and must be triggered manually by users - After each backup execution (whether automatic or manual), the
nextBackupAtfield is recalculated based on thecronExpression. For manual-only schedules, this field remainsnull
Backup Schedule Configuration:
Backup schedules support two separate fields for specifying which files to include:
includePaths: An array of exact file or directory paths (raw, absolute paths). These are typically selected through the file browser in the UI and are passed to restic using the--files-from-rawoption with null-separated values.includePatterns: An array of glob patterns for matching files/directories to include. These patterns are passed to restic using the--files-fromoption with newline-separated values.
Both fields are optional and can be used independently or together. When neither is specified, the backup includes the entire source directory.
Manual-Only Schedules:
Backup schedules can be created without a cron expression for manual-only backups:
- When
cronExpressionis empty or null, the schedule displays "Manual only" in the UI - Manual-only schedules have
enabled=falseandnextBackupAt=nullsince they will not run automatically - Users must manually trigger backups for manual-only schedules through the UI
Validation:
- If a schedule has
enabled=true, it must have a validcronExpression - Empty
cronExpressionis only allowed whenenabled=false(manual-only mode) - When creating or updating a schedule, the system validates this constraint and returns an error if violated
Custom Restic Parameters#
Backup schedules support custom restic parameters that allow advanced users to customize restic behavior for specific backup scenarios. This is an advanced feature for users familiar with restic command-line options.
Configuration:
- Stored in the
customResticParamsfield in the database as an array of strings - Configured through an "Advanced" section in the create/edit schedule form
- Users can specify arbitrary restic flags (e.g.,
--ignore-inode --ignore-ctimefor performance optimization,--iexclude-larger-than 500M,--no-scan,--read-concurrency 8)
Implementation:
- Parameters are stored as an array in the backup schedule configuration
- During backup execution, parameters are passed to the
restic.backup()command - Each parameter string is tokenized by splitting on whitespace, and tokens are appended directly to the restic command arguments
- This provides flexibility to pass any restic-supported flags without requiring code changes
Usage Considerations:
Users should be cautious with custom parameters as they can affect backup behavior and compatibility. These parameters are applied in addition to the standard backup configuration and may override or conflict with default settings.
UI:
The create/edit schedule form includes an "Advanced" section with a textarea where users can enter custom restic parameters, one flag per line. The form provides examples and descriptions to guide users in specifying valid flags.
The frequency dropdown in the create/edit schedule form includes the following options:
- Manual only
- Hourly
- Daily
- Weekly
- Monthly
- Custom (cron expression)
Selecting "Manual only" creates a schedule without automatic execution.
UI Components#
Repositories Table#
The repositories table provides a browsable interface for managing repositories with enhanced visual design.
Location: app/client/modules/repositories/routes/repositories.tsx
Visual Design:
- Table Row Styling: Standardized row height of
h-12for consistent appearance across all pages - Hover Effects: Subtle transitions with
hover:bg-white/2background andhover:border-white/10border effects, replacing the previoushover:bg-accent/50 - Typography: Repository names display in monospace font (
font-mono) withtext-strong-accentcolor for better readability and technical aesthetics - Repository Types: Backend types shown with
text-muted-foregroundcolor in monospace styling - Status Display: Status badges shown with centered alignment using color-coded backgrounds
- Managed Badge: Provisioned repositories display a "Managed" badge next to their name, indicating they are controlled by the provisioning configuration file
- Footer Formatting: Repository count displayed with
font-monostyling, with the count number highlighted usingtext-strong-accent font-boldfor emphasis
Interactive States:
The dashboard redesign improved interactive states and visual consistency throughout the repositories interface, providing better feedback for user interactions with smooth transitions and refined hover effects.
CompressionStatsChart#
The CompressionStatsChart component visualizes repository compression statistics in a card format.
Location: app/client/modules/repositories/components/compression-stats-chart.tsx
Displayed on: Repository info tab (rendered within a grid layout alongside the main repository settings card)
Purpose: Provides visual feedback on repository storage efficiency and compression effectiveness
Metrics displayed:
- Stored Size: Actual size of data on disk after compression
- Uncompressed Size: Original data size before compression
- Space Saved: Both percentage and absolute bytes saved through compression
- Compression Ratio: How much the data has been compressed (e.g., 2.5x)
- Snapshots Count: Total number of snapshots with compression progress percentage
Data Loading:
- Statistics are fetched from the
/api/v1/repositories/{shortId}/statsendpoint using React Query (getRepositoryStatsOptions) - Returns stored statistics from the database, not computed on-demand
- Empty statistics are displayed when a repository has no stats yet (before first backup or after configuration changes)
- The component displays a message indicating stats will be populated after the first backup or manual refresh
Manual Refresh:
- Includes a refresh button (RefreshCw icon) that allows users to manually update statistics
- Clicking the refresh button calls the
POST /api/v1/repositories/{shortId}/stats/refreshendpoint - Shows a loading spinner while refreshing
- Displays success/error toasts after refresh completes
- Users should refresh statistics after operations that change repository contents to see updated metrics
Implementation:
- Handles loading states while fetching statistics
- Displays error messages if statistics cannot be retrieved
- Shows empty state with guidance when no statistics are available yet
- Formats byte sizes and percentages for readability using the
ByteSizecomponent - Uses safe number parsing to handle edge cases and prevent display of invalid values
LocalRepositoryForm#
The LocalRepositoryForm component provides a form interface for creating or importing local repositories.
Location: app/client/modules/repositories/components/repository-forms/local-repository-form.tsx
Context-Aware Path Handling:
The form displays different descriptions and UI elements based on the isExistingRepository flag:
Creating New Repositories:
- Description: "A unique subdirectory will be created inside this directory to store the repository."
- Path display includes a visual preview with
/{unique-id}suffix to indicate automatic subdirectory generation - Example: User enters
/my/path, displayed as/my/path/{unique-id}
Importing Existing Repositories:
- Description: "The exact path to your existing repository."
- Path display shows only the specified path without modification
- Example: User enters
/my/path/existing-repo, displayed as-is
This visual feedback helps users understand the path handling behavior before creating or importing a repository.