Phase 2b: Resource Management#

This phase introduces full support for file-based resources within CipherSwarm. These include wordlists, rule files, masks, and other artifacts required for hashcat operations. All uploads are stored in MinIO using presigned URLs, with metadata tracked in the database.

This phase was previously Phase 4 but has been moved up to Phase 2b to enable Web UI support in Phase 3.

See Phase 2 - Part 2 for UI endpoint scaffolding. This document defines the backend service implementation.

Table of Contents#

• Phase 2b: Resource Management
  • Table of Contents
  • Goals
  • Required Endpoints (Web UI)
  • Access Control
  • Validation & Deduplication
  • Resource Lifecycle
  • Notes
  • Test Cases
  • Implementation Tasks

Goals#

• Integrate MinIO object storage for resource file handling
• Store file metadata (size, line count, hash, type, etc.) in the DB
• Issue presigned upload and download URLs to users and agents
• Detect duplicates using content hashing
• Support access control, sensitivity tagging, and validation
• Allow inline editing of small text-based files (e.g. masks, rules)
• Allow linking resources to attacks with GUID references
• Ensure uploaded resources never orphan or bypass backend registration

Required Endpoints (Web UI)#

Access Control#

Validation & Deduplication#

• Use MD5 or SHA-256 hash to detect duplicates (stored in DB)
• Prevent multiple uploads of identical files
• Raise error if file is already linked to an active attack or campaign - Allow user to link to the existing resource rather than reuploading

Resource Lifecycle#

1. Upload Registration

   • Client posts metadata (name, resource_type, project_id, etc.)
   • Server returns presigned URL for upload
   • DB record created with status = pending

2. Client Uploads File

   • PUT request directly to MinIO using presigned URL

3. Metadata Refresh

   • Client triggers POST /refresh_metadata
   • Server:
     • Downloads file in thread
     • Computes hash, byte size, line count
     • Marks status = active

4. Validation (Optional)

   • Client may preview or validate lines via GET /lines

5. Link to Attack

   • Attack refers to resource by DB id

Notes#

• Presigned URLs should expire after 15 minutes
• Validate that uploaded files match declared resource_type
• If the resource does not successfully upload, the resource should be deleted from the database.
• Inline editing should only be allowed if:
  • Size < 1MB
  • Line count < 5,000 (configurable)
• Deletion must only be allowed if resource is not referenced by any attack

Test Cases#

• Upload a valid rule file and validate it line by line
• Upload a duplicate wordlist and reject it
• Upload large file and verify preview/edit is disabled
• Attempt to delete linked resource → should fail
• Upload, then link to attack, then unlink, then delete → should succeed

Implementation Tasks#

• Use minio-py for all MinIO access (lightweight and sufficient for presigned flow) task_id:minio.minio_py_support
  • All blocking operations must use asyncio.to_thread(...) inside FastAPI
• Add MinioContainer from testcontainers.minio support to enable integration tests for MinIO-based services task_id:testcontainers.minio_support (see Testcontainers MinIO Support) - This is partially implemented in tests/conftest.py, but needs to be fully implemented and should be tested.
• Create StorageService class task_id:minio.storage_service - This is now fully implemented and tested in app/core/services/storage_service.py.
  • Stub out the class and functions and add tests for them. test_id:minio.storage_service_stub - Fully implemented and tested.
  • presign_upload(bucket, key)
  • presign_download(bucket, key)
  • get_file_stats(bucket, key) → byte size, line count, checksum
• Create Pydantic + SQLAlchemy models for AttackResourceFile task_id:minio.attack_resource_file_model
  • Fields: name, resource_type, guid, bucket, key, size_bytes, line_count, checksum, sensitivity, project_id
  • Enum: resource_type: [word_list, rule_list, mask_list, charset, dynamic_word_list]
• Add upload registration endpoint: task_id:minio.upload_registration_endpoint - This is partially implemented (see app/api/v1/endpoints/web/resources.py as upload_resource_metadata(...)), it needs to be wired to use the StorageService, and needs to be fully tested. The upload registration endpoint is used to register a new resource with the database and return a presigned URL for the client to upload the file to. A background task should be created to verify the file was uploaded successfully after a configurable amount of time (default 15 minutes, set in app/core/config.py, tests should override this to a very short value) or delete the resource if it is not uploaded after that time, unless the client notifies the server that the file was uploaded successfully via the upload verification endpoint.
  • POST /api/v1/web/resources/ returns:
    • DB record ID
    • presigned PUT URL
    • file key format: resources/{resource_id}/{filename}
  • Ensure that appropriate tests are added to tests/integration/web/test_web_resources_storage.py to test the upload registration endpoint, with one success and two failure tests.
• Add a new field to the AttackResourceFile model: is_uploaded: bool - This field should be set to True when the file is uploaded successfully and False by default. If the field is true, the verify_upload_and_cleanup(...) task should log that the file was uploaded successfully and exit without deleting the resource.
• Add upload verification endpoint: task_id:minio.upload_verification_endpoint - The upload verification endpoint allows the client to notify the server that the file was uploaded successfully. This is used to update the resource metadata and set the is_uploaded field to True.
  • POST /resources/{id}/uploaded re-fetches size/lines/checksum and updates the resource metadata.
  • Ensure that appropriate tests are added to tests/integration/web/test_web_resources_storage.py to test the upload verification endpoint, with one success and two failure tests.
• Add metadata refresh: task_id:minio.metadata_refresh
  • POST /resources/{id}/refresh_metadata re-fetches size/lines/checksum
  • Run in thread-safe context
  • Ensure that appropriate tests are added to tests/integration/web/test_web_resources_storage.py to test the metadata refresh endpoint, with one success and two failure tests.
• Audit app/api/v1/endpoints/web/resources.py to ensure that all endpoints that interact with file-backed resources are using the StorageService to access the file content.
  • Ensure that appropriate tests are added to tests/integration/web/test_web_resources_storage.py to test the content validation endpoint, with one success and two failure tests.
  • Many endpoints currently have tests in tests/integration/web/test_web_resources.py and should rely on the web-based endpoints to test the content validation endpoint.
  • The resource_service should be updated to use the StorageService if it is not already.
  • Check the following endpoints:
    • get_resource_content(...) - task_id:minio.get_resource_content_endpoint
    • list_wordlists(...) - task_id:minio.list_wordlists_endpoint
    • list_rulelists(...) - task_id:minio.list_rulelists_endpoint
    • list_resource_lines(...) - task_id:minio.list_resource_lines_endpoint
    • add_resource_line(...) - task_id:minio.add_resource_line_endpoint
    • update_resource_line(...) - task_id:minio.update_resource_line_endpoint
    • delete_resource_line(...) - task_id:minio.delete_resource_line_endpoint
    • get_resource_preview(...) - task_id:minio.get_resource_preview_endpoint
    • list_resources(...) - task_id:minio.list_resources_endpoint
    • upload_resource_metadata(...) - task_id:minio.upload_resource_metadata_endpoint
    • audit_orphan_resources(...) - task_id:minio.audit_orphan_resources_endpoint
    • get_resource_detail(...) - task_id:minio.get_resource_detail_endpoint
    • upload_resource_content_chunk_complete(...) - task_id:minio.upload_resource_content_chunk_complete_endpoint
• Add orphan audit: task_id:minio.orphan_audit - This is partially implemented (see app/api/v1/endpoints/web/resources.py as audit_orphan_resources(...)), it needs to be wired to use the StorageService, and needs to be fully tested. The orphan audit is used to ensure that no MinIO objects exist without matching DB rows.
  • Ensure no MinIO objects exist without matching DB rows