HFlow Open source Star on GitHub
Hebbian Robotics · YC S26

Open-source data infrastructure for Physical AI.

HFlow is an open-source SDK for scalable multimodal data pipelines in robotics and Physical AI. It brings production-grade tools and practices—typically developed inside large robotics data teams—to anyone building with robot data.

Star HFlow on GitHub See how HFlow works

Show your support by starring the repository and sharing HFlow with your network.

episode / kitchen_042.mcap pipeline · a41c9f27
/wrist_cam /joint_states /actions
Running Transform topic-group chunks
Ready QC gate 12 measurements
Ready Enrich labels + artifacts
Durable output Curated episode
Traceable
formatMCAP
catalogParquet
provenanceStamped
statusHealthy
One episode carries synchronized streams through processing, quality checks, and curation.
Apache 2.0 Python SDK MCAP episodes Airflow orchestration Parquet catalog

The data lifecycle

From raw recordings to reliable datasets.

HFlow turns multimodal recordings into traceable, quality-tagged episodes and a queryable catalog. Each stage produces an output you can inspect, reproduce, and own.

  1. 01

    Collect

    Land synchronized video, state, action, and sensor streams as one MCAP episode.

    episode_0042.mcap
  2. 02

    Process

    Run your transformations, quality checks, labels, and enrichments as a visible pipeline.

    transform → check → enrich
  3. 03

    Curate

    Query metadata and quality evidence across the corpus without loading the recordings.

    SELECT … FROM episodes
  4. 04

    Deliver

    Produce version-pinned manifests and curated episodes for downstream applications.

    manifest.parquet

Your processing code stays yours

Add HFlow around the code you already trust.

Write plain Python functions for transformations, checks, labels, and enrichments. HFlow records the inputs, outputs, versions, and measurements around each step.

  • Test the complete pipeline locally without Docker.
  • Schedule the same steps as an Airflow 3 DAG.
  • Keep every durable output in standard formats.
Read the quickstart on GitHub
pipeline.py 
import hflow

app = hflow.App("kitchen-pipeline", data_root="./data")

@app.check(critical=True)
def camera_blackout(episode: hflow.Episode) -> hflow.CheckResult:
    frame_stats = hflow.ffmpeg.frame_stats(episode.video("wrist_cam"))
    return hflow.CheckResult(
        measurements={"black_pct": frame_stats.black_frame_pct},
        verdict=frame_stats.black_frame_pct < 50.0,
    )

app.test("episode_0042.mcap")
uv add hflow

Production-grade practices, built in

Know what happened to every episode.

HFlow treats pipeline history, measurements, and versions as durable data—not information trapped inside logs.

PROV

Traceable provenance

Each processed episode records the pipeline and tool versions that produced it.

pipeline_versiona41c9f27b3d8
EVID

Evidence, not verdicts

Checks preserve reusable measurements so each dataset can apply its own thresholds.

black_frame_pct0.18
DAG

Visible orchestration

Rendered Airflow DAGs show task state, logs, retries, dependencies, and reruns.

SQL

Reproducible curation

DuckDB queries write version-pinned manifests with explicit coverage information.

manifestfolding_v7.parquet

A pipeline you can see

Develop locally. Operate with Airflow.

Test the full pipeline in-process, then generate an Airflow 3 DAG for scheduled runs. The same processing functions move from development to durable orchestration.

HFlow renders the ingestion pipeline as an Airflow DAG.

Standard formats at every boundary

Own the data. Replace the pieces.

HFlow connects formats and tools that robotics and data teams already use. Your recordings, catalogs, and manifests remain accessible without HFlow.

MCAP
Canonical multimodal episode
video · state · action
Airflow
Scheduled pipeline execution
DAGs · logs · retries
Parquet
Queryable episode catalog
metadata · evidence · versions
DuckDB
Corpus-wide curation
SQL · coverage · manifests

Project questions

What to know before you start.

HFlow is pre-v1. The core lifecycle works end to end, and you can try it locally today.

Read the complete FAQ
Who is HFlow for?

HFlow is for robotics and Physical AI teams that process multimodal recordings and need traceable, repeatable data pipelines.

Do I need robot hardware?

No. The quickstart generates a small multimodal episode, so you can run the full lifecycle without hardware, Docker, or Airflow.

Does HFlow replace my processing code?

No. Your transformations, checks, labels, and enrichments remain plain Python functions. HFlow adds orchestration, storage, versioning, and curation around them.

Can I run HFlow on my own infrastructure?

Yes. Run a single workspace with Docker Compose or deploy the generated DAG bundle into an Airflow 3 environment that you operate.

How can I contribute?

Contribute code, documentation, examples, compatibility work, bug reports, or design feedback. The contribution guide lists the current priorities.

Open source, built together

Help build the future of robotics.

Our goal is to grow an open-source community where anyone can participate in building better robotics data infrastructure. Star HFlow to show your support and help more developers discover the project.

Star HFlow on GitHub
Contribute to HFlow Join the Discord community