Object Store File Iteration#

Mount file iteration in agenta uses object-store listing APIs — not filesystem walk calls — because mount contents live directly in S3-compatible object storage (SeaweedFS in dev, S3/R2/MinIO in production). The FUSE mount (geesefs) gives agents a filesystem view, but all server-side listing, counting, archiving, and browsing goes directly against the object store via three primitives in ObjectStore and orchestration logic in MountsService.

Three Listing Primitives#

All three methods are on ObjectStore in api/oss/src/core/store/storage.py 1:

list_objects_page accepts start_after and max_keys (default 500) and stops after max_keys objects, making it safe against huge mounts. 1

File Tree Traversal#

_list_pruned_files — git-aware tree descent#

_list_pruned_files in MountsService walks the tree level by level using list_objects_shallow, running sibling directories concurrently (bounded by _LIST_CONCURRENCY = 24). This is the git-aware path 2:

• A frontier of prefixes is listed concurrently each level; results feed the next level's frontier.
• .git directories and .gitignore-matched directories are pruned at the store layer, so dependency dumps like node_modules are never enumerated at all.
• .gitignore specs are read before their level's children are pruned — each level's repo rules apply to that level's subdirectories.
• A visited set prevents re-listing a prefix, guarding against empty-folder markers (trailing-slash objects) that the store surfaces as subdirectory entries.
• The optional cap parameter implements an early exit: once len(kept) >= cap, traversal stops and returns truncated=True. 3

Flat list_objects_v2 path#

Non-git-aware code paths call list_objects_v2 directly for a full, unfiltered recursive listing. Folder markers (keys ending in /) are filtered client-side. Used for the browse tree, archive work list, and folder deletes. 4

Bounded Count-Only Scanning (_COUNT_CAP)#

_COUNT_CAP = 20000 caps count-only (limit=0) requests 5. Two branches:

• git-aware: _list_pruned_files with cap=_COUNT_CAP — descends level by level, stops when 20 000 files are found.
• raw: pages through list_objects_page with max_keys=max(cap, 200) per page, accumulating file objects (non-/ keys) until the count exceeds the cap or has_more=False. 6

Both paths set truncated=True and let the caller report total_capped=True, so the UI shows "N+" without ever blocking on a full enumeration. Prior to PR #5411, the raw branch walked every object with no cap and always returned truncated=False. 7

Archive Iteration and Zip-Slip Prevention#

build_archive_work_list resolves all mounts eagerly (before streaming starts), so errors surface as real HTTP 404/503 rather than silently truncating a 200 response. It calls list_objects_v2 and builds (zip_path, storage_key, size, mtime) tuples. 8

Because store keys are written by signed-credential callers, they can contain path traversal patterns. Three helpers guard archive extraction 9:

• _zip_segments(path) — splits on both / and \ (Windows extractors treat backslash as a separator).
• _has_unsafe_zip_segment(segments) — detects empty, ., or .. segments.
• _safe_zip_segments(path) — strips unsafe segments from zip entry names.

When _has_unsafe_zip_segment fires on a store key, the entry is skipped with a warning rather than rewritten. Rewriting a/../report.txt would alias onto the real a/report.txt and overwrite it on extraction. 10

iter_archive_members consumes the precomputed work list, keeping _ARCHIVE_READ_CONCURRENCY = 8 object reads in flight at once (ordered prefetch via asyncio.deque). On client disconnect, in-flight tasks are cancelled to avoid orphaned reads. 11

Symlinks#

Object stores have no native symlink concept. The FUSE layer (geesefs) handles symlink resolution transparently for agents reading/writing via the mounted filesystem. Server-side listing treats all keys as plain objects; symlinks written through the FUSE mount appear as regular objects under their key. There is no special symlink handling in MountsService or ObjectStore.

Key Sources#