What a neural object actually contains

A neural object is the unit of memory in Realm (earlier called NTL). It is a directory. You can ship it on a USB stick. You can mount it in a container. You can diff two versions in git. The schema is a typed Pydantic model with a JSON Schema mirror, and every bundle round-trips through both before it leaves the pipeline.

This post walks the schema.

The shape of a bundle

neural_objects/<object_id>/
├── manifest.json
├── asset/
├── qualification/
├── task_dna/
├── transfer_report.json
├── human_demonstrations/
├── videos/
├── reproduction/
└── schema/

The directory is the schema’s source of truth in physical form. Each subdirectory has typed contents, listed below.

The seven fields

A neural object manifest has seven top-level fields. We resisted the urge to make this number bigger. Each field earns its place.

1. asset

The thing itself. An OpenUSD file with physics, articulation, and materials authored in. A textured visual mesh (PLY) for human inspection and lightweight rendering. A collision mesh derived through STRATEGY_THIN_DECOMPOSITION with bounded hull counts.

We use OpenUSD because every modern simulator can read it. We use PLY because every modern viewer can read it. There is no proprietary format here.

2. qualification

The 16-test gauntlet result (T1–T16): pass/fail per test, with a numeric margin where one is meaningful. Plus a separate physics_calibration.json recording mass, friction coefficients, joint static friction, and surface friction combine mode.

A separate post covers the gauntlet itself.

3. task_dna

The most distinctive part of the schema. A task DNA record is a structured trace of one execution attempt:

• which task was attempted (open_drawer, press_trigger, lift_lid)
• which robot (Franka 7-DOF, Allegro 16-DOF, Robotiq 2F-85)
• which end-effector
• the trajectory (waypoints with timestamps, gripper states, optional actuate blocks)
• what the physics did: peak grip force, sustained force, joint travel, bbox state changes
• the source flavor: joint_actuate, real_grip_rl, human_demo, or magnetic_grasp

A bundle can carry many task DNA records. The drawer ships with three. We treat the flavor field as load-bearing because it disambiguates idealization from honest physics. A joint_actuate rollout looks identical on video to a real_grip_rl rollout. The flavor field tells the consumer they aren't the same evidence.

4. transfer_report

For every (task, robot, end-effector) combination supported by a task DNA record, the transfer solver computes a verdict and a reason. A redacted excerpt:

{
  "task": "open_bottom_wide",
  "results": [
    {
      "robot": "franka_panda",
      "verdict": "yes",
      "margin": 1.2,
      "reason": "peak_grip 0.93 N >= force_max_N 0.78 N",
      "source_dna": "task_dna_franka_rl.json"
    },
    {
      "robot": "allegro",
      "verdict": "no",
      "margin": 0.71,
      "reason": "peak_grip 0.41 N < force_max_N 0.58 N",
      "source_dna": "task_dna_franka_rl.json"
    }
  ]
}

The verdict is not a confidence number. It is a gate result with a reason a reviewer can audit. We made this choice deliberately. The cross-robot transfer post argues for the design.

5. human_demonstrations

A first-class field in v0.2.0, promoted from a side artifact in v0.1.0. Each demonstration carries:

• source_video.mp4: the original capture
• hand_pose.json: 21-keypoint MediaPipe trace
• gripper_trajectory.json: retargeted waypoints in the asset’s frame
• replay_validation.json: what happened when those waypoints were played against the asset

We treat human demonstrations as memory, not training data. A captured demo on a drawer is evidence about that drawer. It tells future policies what a human did, and what the physics responded with. Whether downstream consumers use it as training data, validation, or simply documentation is their choice.

6. videos and reproduction

Videos are for the human. Reproduction is for the machine. reproduction/run_task.py plus a small README mean any reviewer can replay the bundle in one command. We test this on every release.

7. schema/

The schema files travel inside the bundle. A neural object handed to you in 2030 still says, definitively, which schema version it conforms to. We learned this the hard way watching ROS bag files lose their type definitions.

What round-trip validation rules out

The Pydantic model has aggressive validators. We didn’t write them all up front; each one came from a real bug.

• A task DNA record without a flavor field fails validation. We added this after a magnetic_grasp rollout was almost shipped as real_grip_rl.
• A transfer report whose source_dna reference doesn’t exist in the bundle fails validation. We added this after a stale report survived a DNA file rename.
• A bundle whose human_demonstrations[*].replay_validation references a robot not declared anywhere fails. This caught two copy-paste errors in one week.
• A qualification record whose physics_calibration.mass_kg differs from the USD authored mass fails. We added this after one drawer’s USD said 4 kg and the calibration record said 1.5 kg.

Validators are a place to encode lessons. Every one of ours has a story.

What we didn’t put in the schema

We were tempted. We resisted.

Affordance latents. Tempting because Splat-MOVER is well-cited and the field expects them. We have a typed stub for them. Until we have a deterministic story for what produces these and what consumes them, the stub remains empty. Better to ship a typed empty field than a populated field nobody trusts.

Reactive policy fragments. Same logic. The schema has the slot. The slot is empty.

Continuous neural updates. Not in the schema. Not on the roadmap for v0.3. We will add it when we have an updater that produces something a reviewer can audit.

Confidence scores everywhere. We have one place for confidence (the transfer report’s margin), and even there it is grounded in a physical quantity, not a softmax. Free-floating confidences are how robotics papers stop being reproducible.

What this format buys you

• Diffability. Two versions of a bundle diff cleanly in git. We use this for asset versioning every day.
• Auditability. Every claim in transfer_report.json traces back to a task DNA record traces back to an asset USD. The full chain is one walk away.
• Portability. The bundle has no implicit dependencies on our pipeline. A team that builds neural objects with a completely different stack can produce a v0.2.0 bundle and ours will accept it.
• Honesty about idealization. The flavor field on task DNA forces the bundle author to declare whether physics was real or labelled-cheat.

We chose format over framework. Frameworks are how you lock customers in. Formats are how you let them out.

Roadmap

• v0.3 will add an affordance_regions field with deterministic geometric heuristics. The schema will not require it.
• v0.3 will add pipeline_gaps.json: a structured record of issues this bundle exposes about the pipeline that built it. Inspired by the recurring pattern that every run surfaces a small list of problems worth fixing.
• v0.4 is reserved for the first version of reactive policy fragments, gated on a working updater. No promised date.

Next: From phone video to a Franka pulling a drawer in seven minutes of training: three worked examples (lighter, controller, drawer) of the format in motion.