feat(content): file uploads with public links #94

Merged

peersky merged 17 commits from feat/operator-file-uploads into dev 2026-09-03 10:59:07 +00:00

Conversation 0 Commits 17 Files changed 13 +4882 -2

peersky commented 2026-09-01 02:28:18 +00:00

Owner

Copy link

No description provided.

peersky added 3 commits 2026-09-01 02:28:18 +00:00

feat(content): file uploads with public links ... deee28dc1d

- Added Ability to store a file and hand out a URL anyone can open, over an agent's own bucket credentials.
- Added Interface `content_blob`: `Blob`, `NatsBlob`, the object metadata encoding and the byte ceiling.
- Added Redundancy-free split: the blob layer sits in the SDK because uploader and server are separate processes that must agree on the metadata keys; naming and quota policy stay with the side that owns storage.
- Added Interface `POST /api/content`, spooling the body to a temporary file while hashing it — the digest names the object, so it must be known before the store is written, and a video cannot be hashed in memory.
- Added Ability to refuse an oversize upload while the body streams, because reading a whole body to discover it was too large is the denial of service the cap exists to prevent.
- Added Ability to sniff the media type from the file's own bytes against an allowlist: the stored type is echoed back as `Content-Type` publicly, so an unrecognised one is refused rather than guessed at.
- Added Interface 507 with `used_bytes` and `quota_bytes` on a full bucket, which a caller can act on, rather than a 500 carrying a string it cannot.
- Added Interface `GET /api/content/{digest}` with `Range`, streaming its windows so a whole-object read never assembles the file in memory.
- Added Interface `DELETE /api/content/{digest}` and `GET /api/content/usage`.
- Added Function `NatsNsedWorker::jetstream`, so a caller holding a worker opens buckets on that connection instead of dialling a second one.
- Improved Ability to answer 503 rather than 404 where `NSED_FILES_BUCKET` is unset, so a misconfigured deployment is distinguishable from a file that was never there.

feat(hls): segment uploaded video so viewers can seek ... 6d26bd34d4

- Added Ability to segment an uploaded video into HLS in the background, storing each segment as its own content-addressed object and a playlist that points at their public URLs.
- Fixed Performance problem this exists for: the object store has no server-side range read, so a viewer scrubbing one large file re-reads it from the start on every drag. Segments are small whole-object reads.
- Added Interface `GET /api/content/{digest}/status` — the uploader's poll, since a transcode outlives the request that started it. Reports `pending`/`ready`/`failed`/`skipped` plus the playlist URL.
- Added Ability to refuse a playlist that names a piece which was not stored: playback failing halfway through is worse than never producing one.
- Added Ability to remove everything already stored when a step fails, because a half-stored segmentation costs the operator quota and nothing references it.
- Added Redundancy: streams are copied rather than re-encoded, since most uploads are already H.264/AAC.
- Added Ability to report `skipped` and keep the original where the host has no ffmpeg — a library user who never asked for a media pipeline gets no failure.
- Added Function `Blob::annotate` / `Blob::notes`, recording a pipeline's result against a stored object.
- Fixed Bug where annotations went through `update_metadata`, which in async-nats 0.47 writes back only name and description and drops the map. A sidecar object holds them now, off the serving path.
- Fixed Bug where concurrent annotations on one digest could lose each other; two uploads of identical bytes share a digest, so that was reachable.
- Fixed Bug where a hung ffmpeg held the only transcode slot forever. Killed after 30 minutes, with stderr drained concurrently so a full pipe is not mistaken for a hang.
- Fixed Bug where queued uploads each held a spool on disk without bound. At most four wait; past that an upload is stored whole and reported skipped.
- Fixed Bug where a playlist URL was offered for a segmentation that had since failed, which fails in the player rather than at the API.

test(hls): cover the real transcode, and stop broker tests skipping in CI ... af9dc88281

- Added Ability to test the actual ffmpeg path: a fixture video is built with ffmpeg, segmented for real, and every URL the resulting playlist hands a player is asserted to resolve to bytes we stored.
- Fixed Redundancy in coverage: a fake transcoder cannot tell us the argument vector we run produces a playlist, which was the one part of the pipeline nothing checked.
- Added Ability for CI to install ffmpeg, so that test runs rather than skipping.
- Fixed Bug where CI ran the workspace without `REQUIRE_NATS`, so every broker-backed test could skip and still report green. The job provides a broker; a skip there means coverage silently stopped.

peersky added 1 commit 2026-09-01 02:35:29 +00:00

refactor(files): group the upload subsystem under one module ... 1c6c13a520

- Improved Interface: `content_blob.rs` and `hls.rs` were two loose files at the crate root carrying one subsystem between them, and are now `files::blob` and `files::hls`.
- Removed Redundancy in the crate root, which already holds twenty flat files — adding a whole subsystem to that pile made it harder to see what belongs together.
- Added Function documentation on the `files` module saying what its two halves are for, and where naming and quota policy live instead.

forgejo-actions changed title from feat/operator-file-uploads to feat(content): file uploads with public links 2026-09-01 02:42:22 +00:00

peersky added 1 commit 2026-09-01 06:24:10 +00:00

fix(files): close the upload findings from the pre-merge audit ... 6443e9920c

- BREAKING CHANGE Interface: `visibility` is a required part of an upload. Absent, it was silently public — publishing a file whose uploader may have meant otherwise, invisibly, until the link was out.
- Fixed Bug where a private upload was segmented into public HLS pieces. Segments are the video, so they inherit the source's visibility and a non-public upload is never queued.
- Fixed Bug where a non-`file` multipart part was buffered whole with no ceiling. The route disables the request-level limit so a video can through, so one huge form field could exhaust the process.
- Fixed Bug where the cursor-pool guard was held across the broker round trip and the offset skip: a match scrutinee's temporaries outlive the arm, so one reader's seek serialised every read in the bucket.
- Fixed Bug where `{digest}` reached the store unvalidated, letting a caller read and delete other objects' annotation sidecars through the reserved prefix.
- Fixed Bug where a spool flush error was discarded, storing a short file under the whole file's digest — the one invariant content addressing rests on.
- Added Ability to refuse ffmpeg any protocol but `file`, so an uploaded container's external references cannot be muxed into published segments.
- Fixed Bug where a cancelled transcode left ffmpeg orphaned, and where probing a wedged binary hung agent startup without bound.
- Fixed Bug where an uploader's filename was stored unsanitised for another process to place in a header.
- Fixed Bug where test buckets accumulated 29 per run against JetStream's reserved storage. Names derive from the test and are recreated, so the count is bounded and a panic cannot skip teardown.
- Fixed Redundancy in three vacuous tests, and gated the real-ffmpeg case behind REQUIRE_FFMPEG as the broker cases are behind REQUIRE_NATS.
- Removed Interface churn: `run_control_plane` keeps its nine-argument form, with uploads behind `run_control_plane_with_uploads`.

peersky added 1 commit 2026-09-01 07:28:30 +00:00

ci: make the ffmpeg step fit a self-hosted runner ... f1d7420a9b

- Fixed Bug where the ffmpeg step was the only `sudo` in the workflow, on a runner whose job container is usually root with no `sudo` binary — it would have failed rather than installing anything.
- Added Ability to short-circuit when the image already carries ffmpeg, so baking it into the runner image costs nothing per run instead of ~45s of apt.
- Added Ability to install through apk as well as apt, and to fail with what to do about it rather than a package-manager error, since the runner image is ours to choose.
- Fixed Redundancy: `REQUIRE_FFMPEG` was added to the test gate but never set in CI, so the transcode test could skip and still report green — the exact hole `REQUIRE_NATS` was added to close.

peersky added 1 commit 2026-09-01 08:39:19 +00:00

Merge remote-tracking branch 'origin/dev' into feat/operator-file-uploads c2da7b738d

peersky added 1 commit 2026-09-01 09:42:16 +00:00

refactor(files): one upload implementation for both surfaces ... 30a570950d

- Added Interface `files::upload`: the spool-and-hash, the media-type allowlist, the size cap, the form walk and the store step, as one module both HTTP surfaces call.
- Fixed Redundancy that a second upload route would have created: a viewer depends on the digest, the stored type and the visibility rule all agreeing, and two copies of that logic would have drifted.
- Added Interface `Refusal`, carrying why an upload was rejected rather than a status code, so each surface answers in its own idiom.
- Added Interface `hls::DERIVED_PREFIX` on the filenames segmenting produces, so a library view can list what someone uploaded rather than the hundreds of pieces one video becomes.

peersky added 1 commit 2026-09-01 14:30:10 +00:00

fix(upload): answer an oversize upload before it is sent ... 221bf8c272

- Fixed Bug where an oversize upload was only caught once enough of the body had arrived to exceed the ceiling, so a long upload ended as a reset connection saying nothing about the limit it crossed.
- Added Function `declared_too_large`, refusing from `Content-Length` before the body is read. Absent on a chunked body, where the streaming limit is still what stops it.
- Added Ability for both upload surfaces to use it, so neither drifts into the confusing failure.

peersky added 1 commit 2026-09-01 16:35:45 +00:00

feat(files): configurable upload spool directory, and a full one reported legibly ... cfa6b7df80

- Added Ability NSED_UPLOAD_SPOOL_DIR points the upload spool at a chosen volume, rather than the platform temp dir which in a container is frequently a tmpfs too small for the video it must hold
- Added Interface ENOSPC while spooling becomes its own refusal answered as 507, so a caller learns the server is out of room rather than that something went wrong
- Fixed Bug spool io errors now name the directory they failed in; the previous bare "no spool file" gave an operator no filesystem to go and look at
- Added Redundancy the spool path is logged at error but kept out of the response body, since a server path is not the caller's to see

peersky added 1 commit 2026-09-01 17:10:38 +00:00

refactor(files): publish the HLS annotation keys the serving side must read ... 454cfd5a6d

- Added Interface HLS_NOTE and PLAYLIST_NOTE move from the agent handler into the hls module as public constants, so a process that serves a playlist can find its digest
- Added Redundancy documented why the playlist digest cannot be derived from the source's: a playlist is content-addressed by its own bytes, so the link between the two exists only as an annotation

peersky added 1 commit 2026-09-01 18:10:43 +00:00

feat(files): parse the segmenting state in one place ... ce2cc95b44

- Added Interface HlsState::from_note and as_note replace the ad hoc string match, so the process that writes the annotation, the one that polls it and the one that builds a playlist link cannot disagree about what a note means
- Added Redundancy an absent or unrecognised note reads as Skipped, which is what is true of every object stored before the annotation existed
- Added Function a round-trip test over every state, since a state lost in the round trip is a video that plays in one surface and not another

peersky added 1 commit 2026-09-01 18:34:07 +00:00

fix(files): deleting a video removes its segments too ... ff5ea7933f

- Fixed Bug deleting a video left its segments and playlist in the bucket, so anyone holding the playlist URL kept streaming it and the objects kept spending the operator's quota with nothing referencing them
- Added Interface derived objects are named under their source as hls/<source digest>/<name>, so cleanup finds them from the digest alone rather than from a manifest that could disagree with the bucket
- Added Ability delete_with_derived removes the derived objects before the source, so an interruption leaves a video that still plays rather than segments nothing can find
- Added Function tests that a deleted video leaves the bucket empty, and that deleting one video does not take another's segments with it

peersky added 1 commit 2026-09-01 20:00:12 +00:00

fix(files): the transcoder's working directory is configurable ... ed008d4e1b

- Fixed Bug ffmpeg wrote its output to the platform temporary directory with no way to change it, which in a container is frequently a tmpfs sized for nothing like a video; filling it surfaced only as hls failed
- Added Interface segment_and_store takes a SegmentSpec rather than eight positional arguments, five of them strings and paths the compiler cannot tell apart, where two swapped would point every viewer's playlist somewhere wrong
- Added Redundancy the agent reuses its upload spool directory for the transcode, since a deployment that gave the upload room meant to give its output room too

peersky added 1 commit 2026-09-01 21:39:26 +00:00

test(files): cover deleting a video and uploading it again ... fc1a570563

- Added Function a test that a re-upload is segmented afresh: content addressing returns the same digest, so a note left by the previous life would be read as this one's and point a player at a playlist deleted with the video
- Added Redundancy asserts the annotations are gone after a delete rather than trusting the sidecar removal, which is the only thing standing between a re-upload and a status that claims ready over a 404

peersky added 1 commit 2026-09-01 22:35:37 +00:00

fix(files): an upload past the ceiling reads as too large, not malformed ... 47e782f3bb

- Fixed Bug the route's body cap surfaces as a read error and both loops reported it as malformed, telling a client its request was badly formed when the one thing it could act on was that it was too big
- Added Interface both loops classify by the status the multipart reader assigns, so a cap hit while parsing the next field and one hit mid-body answer alike

peersky added 1 commit 2026-09-02 19:21:18 +00:00

test(files): name the re-upload isolation test in words the spell check accepts ... 7f8788f78a

- Fixed Bug the test name used "anothers", which the spell check rejects and which had already been corrected once in this crate

peersky merged commit c51898c7b5 into dev 2026-09-03 10:59:07 +00:00

peersky deleted branch feat/operator-file-uploads 2026-09-03 10:59:07 +00:00

peersky referenced this pull request from a commit 2026-09-03 10:59:08 +00:00

Merge pull request 'feat(content): file uploads with public links' (#94) from feat/operator-file-uploads into dev

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

release

Automated release PR

No labels

release

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

peeramid-labs/quorum-rs!94

No description provided.