Wire protocol

OpenGrid protocol v1

An implementer’s map to node enrollment, capability discovery, inference streams, receipts, and the retained pipeline contract.

The canonical, field-level specification is PROTOCOL.md at the repository root, paired with versioned protobuf definitions under protocol/opengrid/v1. Those files take precedence over this overview.

Version negotiation

Every envelope carries a semantic protocol version. A node opens with its supported minimum and maximum; the coordinator selects the highest mutually supported minor version within the same major. An incompatible major version closes with a typed upgrade-required error. This allows rolling upgrades without coordinating all nodes.

Transport and identity

QUIC is preferred for multiplexed low-latency streams. Secure WebSocket is the fallback when UDP is unavailable. Nodes always connect outbound, which works through most NATs; an authenticated relay handles unreachable direct paths. TLS is mandatory.

Enrollment creates an Ed25519 device identity. Subsequent control messages carry the enrolled identity, and security-sensitive records are signed over their canonical protobuf encoding plus domain-separation bytes.

Message families

ControlEnvelope
├── ClientHello / ServerHello
├── NodeRegistration / RegistrationAck
├── Heartbeat / CapabilityAdvertisement
├── WorkAssignment / WorkCancellation
├── TokenFrame / StreamEnd
├── PipelineAssignment / ActivationFrame / KVCheckpoint
├── UsageReceipt / ReceiptAck
└── ReputationEvent

Unknown fields must be retained by protobuf implementations. Receivers ignore unknown enum values only where the field is explicitly extensible; otherwise they return a typed protocol error.

Usage receipts

A receipt binds the request nonce, model and route, input/output counters, timing, node identity, and ledger idempotency key. The serving node signs it first. The consumer edge validates stream counters and countersigns it. The coordinator rejects stale nonces, mismatched counters, duplicate idempotency keys, and invalid signatures before settlement.

Retained sharded-pipeline contract

Compatibility only: the v1.4 GGML RPC pipeline remains a stable wire contract, but it is frozen and deprecated as the production cluster strategy. Managed NVIDIA clusters will use a separate Dynamo execution-target boundary; these fields are never reinterpreted as Dynamo prefill/decode or KV transfer.

A pipeline assignment gives every stage an ordered layer interval and peer identity. Activation frames are sequenced and flow-controlled independently from control traffic. Pipeline state is request-scoped and must be erased after completion or cancellation.

In v1, a stage dropout aborts the unfinished generation and rebuilds a route from the original prompt. Metering uses a stable request id and attempt numbers so incomplete work cannot be charged twice. The client receives no duplicated tokens: the edge buffers the uncommitted attempt boundary before exposing data.