AgentCall AG-UI extension 0.1

Public base profile and NFC/photo observations.

This public profile defines AgentCall discovery, sessions, text and voice turns, and WebSocket/WebRTC transport for independent agents, with optional NFC and photo observations. It publishes the shared base required for those physical inputs; other AgentCall extensions are outside this profile. The canonical A2A Agent Card extension URI is https://agentcall.smartycode.com/ext/agentcall-agui/0.1 . Agent implementations are outside this publication. No private repository access is needed to implement the messages below.

Version and migration

The extension path and params.schemaVersion: "0.1" identify this contract. The observation and passport payloads also carry schemaVersion: "0.1". The transport bindings remain agentcall-agui-ws-0.1 and agentcall-webrtc-0.1. Additive optional fields may be introduced in 0.1; removals, changed meanings, required fields, or tighter bounds require a new versioned URI and a documented migration. Consumers pin the artifact manifest's source commit and hashes. Published versioned artifacts are retained.

Clients and conformance accept the previous https://agentcall.invalid/ext/agentcall-agui/0.1 identifier as a legacy alias. Producers emit one identifier per card; duplicate recognized declarations are rejected, including mixed canonical/legacy declarations. Existing private producers may continue advertising the legacy URI. Rollback can restore the previous client and legacy producer card without changing observation bytes. An older client that does not recognize the public URI must report an incompatibility; producers serving such clients retain the legacy card until those clients upgrade. This release does not retire the alias.

Discovery

Serve a standard A2A 1.0 Agent Card at /.well-known/agent-card.json and add:

{
  "uri": "https://agentcall.smartycode.com/ext/agentcall-agui/0.1",
  "description": "AgentCall AG-UI extension 0.1: public base and physical observations",
  "required": false,
  "params": {
    "schemaVersion": "0.1",
    "agui": {
      "url": "https://agent.example/agui",
      "transport": "websocket",
      "session": {"resume": false, "replay": false}
    },
    "voice": {"owner": "client"},
    "physicalObservations": {
      "accepted": ["nfc", "photo"],
      "agentRequests": ["nfc", "photo"]
    }
  }
}

accepted and agentRequests are unique arrays of at most two modalities. Every requested modality MUST also be accepted. Omission or an empty array means unsupported. The declaration permits collection; it never grants access to the camera, NFC hardware, persistent memory, or actions.

Use A2A supportedInterfaces only for actual A2A interfaces. The AG-UI endpoint is advertised in this extension because its semantics differ from A2A task transports. This extension does not change A2A request semantics; an A2A-Extensions header does not negotiate the separate AG-UI connection. AG-UI capability negotiation happens in session.start below. Official A2A conformance belongs to the A2A TCK, not these AgentCall-specific fixtures. Unknown required card extensions are enrollment incompatibilities.

Physical inputs are supported only on WebSocket or the exact agentcall-webrtc-0.1 profile. Cards advertising NFC/photo on openai-realtime-webrtc-1, http.sse, realtime.websocket, or another unmapped profile MUST be rejected. Runtime connectors must also suppress physical capabilities and reject sends for unsupported persisted profiles. No implied mapping to a vendor's image API exists in this release.

WebSocket and WebRTC

For WebSocket, derive wss from the card's HTTPS URL. HTTP/WS is allowed only under an explicit local-development policy. Bearer authentication, if the card requires it, uses the upgrade Authorization header, never a URL. Apply HTTPS, DNS/IP, redirect, timeout, credential, and size validation before connecting. Cards are at most 256 KiB; each UTF-8 JSON frame is at most 128 KiB, before parsing. Reject oversized frames and binary input.

For transport: "webrtc", require voice.owner: "agent" and:

{
  "profile": "agentcall-webrtc-0.1",
  "audio": {"codec": "opus", "clockRateHz": 48000, "channels": 1, "direction": "sendrecv"},
  "events": {"transport": "datachannel", "label": "agentcall.events", "protocol": "agentcall-agui-dc-0.1", "ordered": true}
}

This object is params.agui.webrtc. POST the SDP offer as application/sdp to the card URL and accept an SDP answer of the same media type, bounded to 128 KiB. Send AgentCall-Session-ID and Accept: application/sdp headers; do not follow redirects. Use one bidirectional Opus audio track and one ordered reliable data channel with that exact label and subprotocol. Send the same JSON client messages and receive the same server envelopes on that channel. Photos use the bounded JSON observation, never RTP. Resume/replay are false in this WebRTC version; ICE recovery does not create logical session resume. Optional supportsExplicitTurns: true permits the client capability audioTurnControl: {"mode":"explicit"}. audio.turn.start, .commit, and .cancel carry the same integer turn ID; only one audio turn is open, and the client stops its microphone before commit/cancel. RTP carries audio.

Session and input

The first client message MUST be session.start:

{"type":"session.start","sessionId":"proof-1","clientCapabilities":{"physicalObservations":{"modalities":["nfc","photo"]}}}

The server responds with {"sequence":1,"event":{"type":"session.established","resumed":false}}. Each subsequent server envelope has a strictly increasing positive sequence. The client can acknowledge a sequence with {"type":"ack","sequence":1}. The public proof does not require resume support; on disconnect discard session-local images, proposals, and reference memory. Resume is optional and must not be advertised without a separately tested replay implementation.

The usable modality set is the intersection of the card's accepted modalities and this session's client modalities. Missing client capabilities mean the empty set. Every observation MUST fit that intersection. Check before any agent/model use. Agent requests require membership in agentRequests too. Capability support does not imply permission: the user must initiate each collection while the app is in the foreground through a visible control.

{
  "type":"user.observation",
  "turn":1,
  "observation":{
    "schemaVersion":"0.1",
    "observationId":"scan-1",
    "capturedAt":"2026-09-07T10:00:00Z",
    "intent":"identify",
    "object":{
      "objectRef":"urn:agentcall:object:workshop-router",
      "carrier":"localTag",
      "assurance":"unverifiedTag",
      "payloadSha256":"0000000000000000000000000000000000000000000000000000000000000000"
    }
  }
}

The all-zero digest above is illustrative; a producer computes the actual digest from the exact carrier bytes or local alias. At least object or image is required. An object requires negotiated nfc; an image requires negotiated photo; a combined input requires both. turn shares one strictly increasing counter with user.turn; duplicate/replayed turns are rejected without side effects. observationId is bounded correlation data, not authentication. capturedAt is RFC 3339 data, not trusted device time. intent is identify, remember, inspect, or act, and grants no authority.

Normal text input remains {"type":"user.turn","turn":2,"text":"What is this?"}. Optional AG-UI Context entries are {description, value} strings. A client may carry session-local physical focus as bounded reference/assurance facts on later turns, but never repeats the photo or treats facts as instructions.

The server responds to a processed turn with run.started, followed by text.delta with text, optional stable speech.segment with segmentId, text, stable:true, then run.completed. Only one run is active; ordered envelopes provide correlation. Lifecycle details and bounds are in the published event schema. failure carries bounded code/message and must not leak credentials or echo image bytes. Invalid input must not create memory/actions. response.cancel cancels the active run and pending collection/proposal. session.end cancels pending work and closes the session. Unknown additive fields and unknown event types never grant capabilities or execute actions; clients ignore them or report an unsupported extension.

Why this is a versioned client input, not an upstream CUSTOM event

Pinned AG-UI release release/2026-07-31, commit bb1c2afddb4880309879b9564cfb3a635a5da4eb, has TypeScript core 0.0.57 and Python 0.1.19. Its official CUSTOM event is {type:"CUSTOM",name,value}; its separate RunAgentInput carries messages, context, state, and forwardedProps. It defines no NFC/photo WebSocket input message.

user.observation is explicitly an AgentCall 0.1 client-to-agent input binding. It preserves shipped clients and adds turn ordering, modality and interaction correlation before model input. It is not an upstream event. The agent adapter may map the validated observation into a run's bounded context or multimodal input under its own application policy; it must not accept arbitrary tag text as prompts or forwarded tool arguments. A future CUSTOM/event or other transport adapter requires its own versioned mapping. The normalized server event names in this binding are AgentCall names, too.

Pinned upstream definitions:

Agent-requested collection

After negotiation, an agent may emit interaction.requested with one opaque interactionId, kind:"physical.nfc.scan" or "physical.photo.capture", bounded prompt and spokenPrompt, options:[], and allowsFreeform:false. The client presents a visible scan/capture action; it never silently opens the camera or reader. Only one request is pending. The user's resulting user.observation includes the exact interactionId and requested modality. An unmatched, stale, cancelled, or wrong-modality response is rejected with no action. A valid response consumes the request exactly once and emits interaction.resolved with outcome:"answered". Cancellation emits outcome:"cancelled"; session close clears the pending request. An unrelated scan without the pending ID cannot satisfy it.

Security, photos, and exact object lookup

Treat tags, photos, passports, timestamps, and every supplied field as untrusted. NFC tags are writable labels, not proof of identity, ownership, or permission. signedTag is reserved for a separately verified cryptographic carrier; do not infer it from a string. No tag-supplied code, URL, instruction, credential, or action is executed merely because it was scanned.

Ordinary/empty hardware tags map locally to an opaque keyed alias; raw NFC hardware IDs never leave the client. A localTag digest covers the alias, not the hardware ID. Device-local aliases need not match on other devices; portable NDEF references are separate. Receivers use exact objectRef equality, never prefix, fuzzy, case-folded, or embedding matches for identity. Never automatically fetch an HTTPS object reference. An explicitly authorized resolver must apply its own HTTPS/SSRF/redirect/body-size/time policy.

An image is image/jpeg, strict base64, at most 61,440 decoded bytes and 1,024 pixels on either dimension. The producer strips metadata and derives byteCount, sha256, pixelWidth, and pixelHeight from the final JPEG. The receiver verifies byte count/hash and actual JPEG dimensions before use; JSON Schema alone does not verify encoded-content annotations or digests. Bound image decoding and reject malformed data, forged dimensions, and oversized images. Images are ephemeral session input, excluded from diagnostics and subsequent context. End/cancel/failure releases retained buffers. Photo capture consent does not authorize storing or forwarding it.

An unknown object starts a teaching conversation. Stage a proposal tied to the exact object, show what will be remembered, and obtain a separate explicit confirmation before a persistent write or consequential action. A scan, intent:"remember", an image, or generic assent without a current proposal is insufficient. Replacement proposals need fresh confirmation; switching objects invalidates an old proposal. Enforce agent/user storage isolation. The optional passport is a bounded reference projection, not private history or credentials; every action recipe requires requiresConfirmation:true.

Independent reference proof

The independent AgentCall reference agent's opt-in developer profile proves this contract without importing gateway, TGP, iOS, or protocol implementation code. Its ordinary public review profile remains text-only. The proof uses at most 32 session-local object labels, each at most 256 characters; no user content is written to disk. It is protocol behavior evidence, not physical NFC/camera acceptance or a production memory store.

The reference implementation's wire tests exercise: discover the card; start a session with modalities; scan an unknown object; send teach object Workshop router; verify no stored association yet; send confirm object; rescan the same reference and receive the label. A different reference remains unknown. cancel teaching cancels a proposal. A new scanned object invalidates it. Requests scan an object and take a photo exercise collection messages, correlation, wrong-modality rejection, and single consumption. Actual hardware collection requires separate device acceptance. The public golden trace is a synthetic protocol example; its schema/order checks alone cannot prove an agent's memory or confirmation behavior. The independent reference-agent wire tests provide that behavioral evidence. Implementers must test these outcomes against their own agents, without copying the reference's storage/model design.

Public artifacts

All relative paths below resolve from https://agentcall.smartycode.com/ . The schemas are a generated public projection of the canonical contracts, with only the required dependency closure included. Schema omissions do not publish or authorize other private extensions.

schemas/agentcall-card-0.1.schema.json: public discovery ingest profile.
schemas/agentcall-agui-ws-client-0.1.schema.json: client input messages.
schemas/agentcall-events-0.1.schema.json: normalized server envelopes.
schemas/agentcall-physical-observation-0.1.schema.json: object/photo input.
schemas/agentcall-object-passport-0.1.schema.json: portable reference data.
protocol/0.1/physical-conformance.json: positive/negative extension fixtures.
protocol/0.1/manifest.json: release version, canonical source commit, upstream pins, exact public file inventory, and SHA-256 hashes.

Use the official A2A conformance tooling for the base Agent Card. The AgentCall fixtures additionally check URI migration, modality/profile gates, payload integrity, and the documented physical interaction sequence.