The credentialed capture & annotation pipeline that telemetry can't replace.
Kindly is the credentialed capture, four-layer annotation, and access pipeline — plus the standard schema — for surgical physical AI. The data stays hospital-owned; we operate as the rights-clean processor and annotation/access layer, not a reseller of patient data. We start with non-patient workflows — sterile processing, OR turnover, instrument-tray assembly, and sim/cadaver/vet — where consent friction is minimal, so we can capture real episodes now. The differentiator isn't phase recognition — open corpora already do that. It's the non-telemetry workflow context: interruptions, rework, deviations, and handoffs, captured under credentialed clinical access and labeled to clinical standards.
Credentialed capture of non-telemetry workflow data
Robot telemetry tells you what a machine did. Open robotics corpora (for example NVIDIA's Open-H) give you scale on generic manipulation. Neither contains the layer that makes real clinical work real: the human workflow around the procedure — who handed what to whom, where the count came up short, what was reworked, and which deviations from protocol occurred. That layer is invisible to sensors and absent from scraped video. Capturing it requires credentialed access to the clinical environment and clinical judgment to label it. That is the wedge — and we start it in non-patient workflows (sterile processing, OR turnover, instrument-tray assembly, sim/cadaver/vet), where there is little to no PHI or consent friction, so episodes can be captured now.
Hospital-owned, processor-operated. We don't own or resell the data. The institution owns its dataset; Kindly is the rights-clean processor and annotation/access layer — a CRO/RLHF-style services model with optional co-contribution — so labs and OEMs reach rights-clean, policy-grade supervision without a startup claiming to commercialize patient data.
Why us: our founding team includes a surgical RN and a hospital robotics coordinator. That credentialed clinical team is how we operate the capture pipeline, earn institutional access, and annotate with clinical fidelity — not how we resell what we see.
The four-layer annotation taxonomy
Four independent label tracks over the same timeline — the standard schema a partner can consume in any subset. L1–L2 are the conventional surface; L3–L4 are where the policy-learning value concentrates.
Workflow phase / step
A two-level hierarchy over the workflow timeline — phases decomposed into ordered steps. This is the conventional surface that existing surgical-video datasets stop at.
Action triplets
<actor-role, verb, object> intervals with start/end timestamps. The first slot is who acts (tech, nurse, assist, robot, surgeon), because in workflow automation who acts is a control variable.
Instrument / object state
Per-object tracks that persist across an episode — location zone, sterility state, and count events. This is what turns a video into a manipulable world model a policy can reason over.
Workflow-context events
Typed events for the things that make clinical work real: interruptions, corrections / rework, protocol deviations, and handoffs. This is the differentiating layer and the hardest, most valuable supervision to collect.
A policy trained only on clean runs learns the happy path and fails the moment reality diverges. L4 explicitly localizes the recoveries, corrections, and deviations — exactly the transitions a robust policy must imitate, and exactly what a classification corpus structurally cannot provide.
- Negative & recovery examples — the rare, expensive supervision a robust policy needs.
- Demonstration-quality filtering — tell a clean demo from a salvaged one; filter, down-weight, or train recovery on precisely those segments.
- Reward & constraint signal — deviations and sterility-breach events are directly usable as negative reward or constraint-violation labels for safe-policy and offline-RL methods.
RLDS / LeRobot-compatible serialization
Every episode serializes natively to the RLDS step/episode model used by Open X-Embodiment and round-trips to LeRobot. Your existing OXE / LeRobot pipeline ingests episodes in the schema without a custom loader.
Sidecar is the source of truth
Versioned & auditable
Inspect the schema yourself
Two fully-worked sample episodes spanning two workflow families — an SPD tray-reassembly and an OR instrument-count — each with all four label layers (L1 phase/step, L2 action triplets, L3 instrument-state tracks, L4 workflow-context events including exceptions, corrections, and handoffs) plus a derived RLDS step excerpt. Load them, validate them, see exactly what an episode looks like.
Synthetic examples conforming to schema v0.1 — no real patient or clinical data. Every value is fabricated for illustration; no clinical capture has occurred.
Validate your own capture
We publish the episode format as a machine-readable JSON Schema (draft 2020-12) so you can validate your own captured episode against ours before any data-sharing agreement exists. The schema is permissive by design — a superset of our fields still validates — so it interops rather than locks you in.
The quickest path is our open CLI — it bundles the v0.1 schema, so one command validates an episode against the standard with no extra files to download:
# Easiest: the published CLI (bundles the v0.1 schema)
pip install foodforthought-cli
ate episode validate your-episode.json
# Start from a valid skeleton, then fill it in:
ate episode scaffold --out episode.jsonfoodforthought-cli is on PyPI (v0.5.0); the command is offline and needs no Kindly account. Prefer to wire the schema into your own pipeline? Download it and validate with ajv-cli or Python instead:
# Node (ajv-cli)
npx ajv-cli@5 validate --spec=draft2020 -c ajv-formats \
-s kindly-episode.schema-v0.1.json -d your-episode.json
# Python (jsonschema)
import json, jsonschema
from jsonschema import Draft202012Validator
schema = json.load(open("kindly-episode.schema-v0.1.json"))
Draft202012Validator(schema).validate(json.load(open("your-episode.json")))Both sample episodes above conform to this schema. The schema is v0.1 and will evolve with first-cohort design partners.
Explore the sample episode
Scrub through synthetic sample episodes step by step — switch between an SPD tray-reassembly and an OR instrument-count workflow to see the schema span two workflow families. Each step shows all four annotation layers over the same timeline — objects, actions, instrument state, and the workflow-context events that are our differentiator. This is the exact data in the JSON above, rendered interactively.
Honest status
This schema is v0.1 and in development. No clinical data has been captured against it yet — that is the point of the design-partner program. Our ingestion, labeling, and lineage pipeline is built and running today on non-clinical surrogate data; the first real episodes — hospital-owned, non-patient workflows first, with Kindly as processor — are what first-cohort partners shape. We are pre-funding, pre-contract, pre-IRB, and pre-capture, and we'd rather be exactly this honest than impressive on paper.
The first cohort launches Q3 2026, capped at three design partners, each shaping the capture protocols and annotation schema around their model's needs. There is no fee to participate.
For foundation-model labs and surgical-robot OEMs evaluating a credentialed, rights-clean capture + annotation pipeline for clinical-workflow supervision. Email taylorm@kindly.fyi with your team, your target use case, and what data you've tried so far.