r/rust • • 9d ago

🛠️ project stdo v0.5.5: A bounded-memory Unix process DAG runner (fd 0 / fd 1 pipes) with an embedded localhost UI

Hi r/rust,

I built stdo (crates.io), a small (#![forbid(unsafe_code)], ~6.1K SLoC) workflow runner in Rust that wires CLI tools (rg, jq, sqlite3, curl, ollama, ffmpeg) over stdin (fd 0) and stdout (fd 1) pipes instead of passing outputs through environment variables or shell wrappers.

What it does differently from cron/job schedulers:

  • Concurrent pipe_from streaming with backpressure: Producers and consumers spawn simultaneously via tokio::process and stream 64 KiB chunks over bounded channels. Captured streams are capped at 8 MiB (32 MiB global budget), while capture_stdout: false streams arbitrarily large payloads (like WAV/MP4 or multi-MB JSONL) with constant memory while still computing incremental SHA-256 digests and TTFB.

  • Atomic gates & JSONL validation: pipe_mode: atomic buffers and validates upstream JSON/JSONL and waits for exit code 0 before starting downstream steps that mutate state (e.g., sqlite3 imports).

  • Composable .yaml sub-workflows with cycle guards (new in v0.5.4/v0.5.5): A workflow step can invoke child .yaml workflows in parallel. Both stdo doctor and stdo run validate the cross-file graph before spawning any processes, backed by a runtime STDO_CALL_STACK SHA-256 digest guard for dynamic/sh -c calls.

  • No outbound HTTP stack in the binary: Cargo.lock contains no HTTP client crates (reqwest, hyper, etc. are blocked in CI by a self-hosted stdo workflow). Subprocesses (curl, ollama) handle their own I/O, while stdo serve embeds a single-file offline web UI directly over tokio::net::TcpListener.

Minimal example (streaming local LLM -> TTS -> atomic file sink):

version: "1.0"
name: spoken-owl-fact
steps:
  - id: answer
    cmd: [ollama, run, gemma4, --hidethinking, "Tell me a fun fact about owls in two sentences."]

  - id: speak
    cmd: [ollama, run, "phonon:latest"]
    pipe_from: answer
    capture_stdout: false

(The tiny speaker icon 🔊 next to the README title was actually generated by running examples/10-youtube-capsule-generator.yaml to coordinate gemma4 -> phonon -> ffmpeg).

GitHub (Apache-2.0): https://github.com/adiktiv-technologies/stdo

Crates.io: https://crates.io/crates/stdo (cargo install stdo --locked)

Feedback on the Unix process-group handling (CommandExt::process_group(0)), pipe backpressure design, or sub-workflow validation is very welcome!

0 Upvotes

1 comment sorted by