# AgentCall AG-UI extension 0.1 *Public base profile and NFC/photo observations.* This public profile defines AgentCall discovery, sessions, text and voice turns, and WebSocket/WebRTC transport for independent agents, with optional NFC and photo observations. It publishes the shared base required for those physical inputs; other AgentCall extensions are outside this profile. The canonical A2A Agent Card extension URI is https://agentcall.smartycode.com/ext/agentcall-agui/0.1 . Agent implementations are outside this publication. No private repository access is needed to implement the messages below. ## Version and migration The extension path and `params.schemaVersion: "0.1"` identify this contract. The observation and passport payloads also carry `schemaVersion: "0.1"`. The transport bindings remain `agentcall-agui-ws-0.1` and `agentcall-webrtc-0.1`. Additive optional fields may be introduced in 0.1; removals, changed meanings, required fields, or tighter bounds require a new versioned URI and a documented migration. Consumers pin the artifact manifest's source commit and hashes. Published versioned artifacts are retained. Clients and conformance accept the previous `https://agentcall.invalid/ext/agentcall-agui/0.1` identifier as a legacy alias. Producers emit one identifier per card; duplicate recognized declarations are rejected, including mixed canonical/legacy declarations. Existing private producers may continue advertising the legacy URI. Rollback can restore the previous client and legacy producer card without changing observation bytes. An older client that does not recognize the public URI must report an incompatibility; producers serving such clients retain the legacy card until those clients upgrade. This release does not retire the alias. ## Discovery Serve a standard A2A 1.0 Agent Card at `/.well-known/agent-card.json` and add: ```json { "uri": "https://agentcall.smartycode.com/ext/agentcall-agui/0.1", "description": "AgentCall AG-UI extension 0.1: public base and physical observations", "required": false, "params": { "schemaVersion": "0.1", "agui": { "url": "https://agent.example/agui", "transport": "websocket", "session": {"resume": false, "replay": false} }, "voice": {"owner": "client"}, "physicalObservations": { "accepted": ["nfc", "photo"], "agentRequests": ["nfc", "photo"] } } } ``` `accepted` and `agentRequests` are unique arrays of at most two modalities. Every requested modality MUST also be accepted. Omission or an empty array means unsupported. The declaration permits collection; it never grants access to the camera, NFC hardware, persistent memory, or actions. Use A2A `supportedInterfaces` only for actual A2A interfaces. The AG-UI endpoint is advertised in this extension because its semantics differ from A2A task transports. This extension does not change A2A request semantics; an `A2A-Extensions` header does not negotiate the separate AG-UI connection. AG-UI capability negotiation happens in `session.start` below. Official A2A conformance belongs to the A2A TCK, not these AgentCall-specific fixtures. Unknown required card extensions are enrollment incompatibilities. Physical inputs are supported only on WebSocket or the exact `agentcall-webrtc-0.1` profile. Cards advertising NFC/photo on `openai-realtime-webrtc-1`, `http.sse`, `realtime.websocket`, or another unmapped profile MUST be rejected. Runtime connectors must also suppress physical capabilities and reject sends for unsupported persisted profiles. No implied mapping to a vendor's image API exists in this release. ## WebSocket and WebRTC For WebSocket, derive `wss` from the card's HTTPS URL. HTTP/WS is allowed only under an explicit local-development policy. Bearer authentication, if the card requires it, uses the upgrade Authorization header, never a URL. Apply HTTPS, DNS/IP, redirect, timeout, credential, and size validation before connecting. Cards are at most 256 KiB; each UTF-8 JSON frame is at most 128 KiB, before parsing. Reject oversized frames and binary input. For `transport: "webrtc"`, require `voice.owner: "agent"` and: ```json { "profile": "agentcall-webrtc-0.1", "audio": {"codec": "opus", "clockRateHz": 48000, "channels": 1, "direction": "sendrecv"}, "events": {"transport": "datachannel", "label": "agentcall.events", "protocol": "agentcall-agui-dc-0.1", "ordered": true} } ``` This object is `params.agui.webrtc`. POST the SDP offer as `application/sdp` to the card URL and accept an SDP answer of the same media type, bounded to 128 KiB. Send `AgentCall-Session-ID` and `Accept: application/sdp` headers; do not follow redirects. Use one bidirectional Opus audio track and one ordered reliable data channel with that exact label and subprotocol. Send the same JSON client messages and receive the same server envelopes on that channel. Photos use the bounded JSON observation, never RTP. Resume/replay are false in this WebRTC version; ICE recovery does not create logical session resume. Optional `supportsExplicitTurns: true` permits the client capability `audioTurnControl: {"mode":"explicit"}`. `audio.turn.start`, `.commit`, and `.cancel` carry the same integer turn ID; only one audio turn is open, and the client stops its microphone before commit/cancel. RTP carries audio. ## Session and input The first client message MUST be `session.start`: ```json {"type":"session.start","sessionId":"proof-1","clientCapabilities":{"physicalObservations":{"modalities":["nfc","photo"]}}} ``` The server responds with `{"sequence":1,"event":{"type":"session.established","resumed":false}}`. Each subsequent server envelope has a strictly increasing positive sequence. The client can acknowledge a sequence with `{"type":"ack","sequence":1}`. The public proof does not require resume support; on disconnect discard session-local images, proposals, and reference memory. Resume is optional and must not be advertised without a separately tested replay implementation. The usable modality set is the intersection of the card's accepted modalities and this session's client modalities. Missing client capabilities mean the empty set. Every observation MUST fit that intersection. Check before any agent/model use. Agent requests require membership in `agentRequests` too. Capability support does not imply permission: the user must initiate each collection while the app is in the foreground through a visible control. ```json { "type":"user.observation", "turn":1, "observation":{ "schemaVersion":"0.1", "observationId":"scan-1", "capturedAt":"2026-09-07T10:00:00Z", "intent":"identify", "object":{ "objectRef":"urn:agentcall:object:workshop-router", "carrier":"localTag", "assurance":"unverifiedTag", "payloadSha256":"0000000000000000000000000000000000000000000000000000000000000000" } } } ``` The all-zero digest above is illustrative; a producer computes the actual digest from the exact carrier bytes or local alias. At least `object` or `image` is required. An object requires negotiated `nfc`; an image requires negotiated `photo`; a combined input requires both. `turn` shares one strictly increasing counter with `user.turn`; duplicate/replayed turns are rejected without side effects. `observationId` is bounded correlation data, not authentication. `capturedAt` is RFC 3339 data, not trusted device time. `intent` is `identify`, `remember`, `inspect`, or `act`, and grants no authority. Normal text input remains `{"type":"user.turn","turn":2,"text":"What is this?"}`. Optional AG-UI Context entries are `{description, value}` strings. A client may carry session-local physical focus as bounded reference/assurance facts on later turns, but never repeats the photo or treats facts as instructions. The server responds to a processed turn with `run.started`, followed by `text.delta` with `text`, optional stable `speech.segment` with `segmentId`, `text`, `stable:true`, then `run.completed`. Only one run is active; ordered envelopes provide correlation. Lifecycle details and bounds are in the published event schema. `failure` carries bounded `code`/`message` and must not leak credentials or echo image bytes. Invalid input must not create memory/actions. `response.cancel` cancels the active run and pending collection/proposal. `session.end` cancels pending work and closes the session. Unknown additive fields and unknown event types never grant capabilities or execute actions; clients ignore them or report an unsupported extension. ## Why this is a versioned client input, not an upstream CUSTOM event Pinned AG-UI release `release/2026-07-31`, commit `bb1c2afddb4880309879b9564cfb3a635a5da4eb`, has TypeScript core 0.0.57 and Python 0.1.19. Its official CUSTOM event is `{type:"CUSTOM",name,value}`; its separate `RunAgentInput` carries messages, context, state, and forwardedProps. It defines no NFC/photo WebSocket input message. `user.observation` is explicitly an AgentCall 0.1 client-to-agent input binding. It preserves shipped clients and adds turn ordering, modality and interaction correlation before model input. It is not an upstream event. The agent adapter may map the validated observation into a run's bounded context or multimodal input under its own application policy; it must not accept arbitrary tag text as prompts or forwarded tool arguments. A future CUSTOM/event or other transport adapter requires its own versioned mapping. The normalized server event names in this binding are AgentCall names, too. Pinned upstream definitions: - https://github.com/ag-ui-protocol/ag-ui/blob/bb1c2afddb4880309879b9564cfb3a635a5da4eb/sdks/typescript/packages/core/src/events.ts - https://github.com/ag-ui-protocol/ag-ui/blob/bb1c2afddb4880309879b9564cfb3a635a5da4eb/sdks/typescript/packages/core/src/types.ts - https://github.com/a2aproject/A2A/blob/173695755607e884aa9acf8ce4feed90e32727a1/docs/topics/extensions.md ## Agent-requested collection After negotiation, an agent may emit `interaction.requested` with one opaque `interactionId`, `kind:"physical.nfc.scan"` or `"physical.photo.capture"`, bounded `prompt` and `spokenPrompt`, `options:[]`, and `allowsFreeform:false`. The client presents a visible scan/capture action; it never silently opens the camera or reader. Only one request is pending. The user's resulting `user.observation` includes the exact `interactionId` and requested modality. An unmatched, stale, cancelled, or wrong-modality response is rejected with no action. A valid response consumes the request exactly once and emits `interaction.resolved` with `outcome:"answered"`. Cancellation emits `outcome:"cancelled"`; session close clears the pending request. An unrelated scan without the pending ID cannot satisfy it. ## Security, photos, and exact object lookup Treat tags, photos, passports, timestamps, and every supplied field as untrusted. NFC tags are writable labels, not proof of identity, ownership, or permission. `signedTag` is reserved for a separately verified cryptographic carrier; do not infer it from a string. No tag-supplied code, URL, instruction, credential, or action is executed merely because it was scanned. Ordinary/empty hardware tags map locally to an opaque keyed alias; raw NFC hardware IDs never leave the client. A `localTag` digest covers the alias, not the hardware ID. Device-local aliases need not match on other devices; portable NDEF references are separate. Receivers use exact `objectRef` equality, never prefix, fuzzy, case-folded, or embedding matches for identity. Never automatically fetch an HTTPS object reference. An explicitly authorized resolver must apply its own HTTPS/SSRF/redirect/body-size/time policy. An image is `image/jpeg`, strict base64, at most 61,440 decoded bytes and 1,024 pixels on either dimension. The producer strips metadata and derives `byteCount`, `sha256`, `pixelWidth`, and `pixelHeight` from the final JPEG. The receiver verifies byte count/hash and actual JPEG dimensions before use; JSON Schema alone does not verify encoded-content annotations or digests. Bound image decoding and reject malformed data, forged dimensions, and oversized images. Images are ephemeral session input, excluded from diagnostics and subsequent context. End/cancel/failure releases retained buffers. Photo capture consent does not authorize storing or forwarding it. An unknown object starts a teaching conversation. Stage a proposal tied to the exact object, show what will be remembered, and obtain a separate explicit confirmation before a persistent write or consequential action. A scan, `intent:"remember"`, an image, or generic assent without a current proposal is insufficient. Replacement proposals need fresh confirmation; switching objects invalidates an old proposal. Enforce agent/user storage isolation. The optional passport is a bounded reference projection, not private history or credentials; every action recipe requires `requiresConfirmation:true`. ## Independent reference proof The independent AgentCall reference agent's opt-in developer profile proves this contract without importing gateway, TGP, iOS, or protocol implementation code. Its ordinary public review profile remains text-only. The proof uses at most 32 session-local object labels, each at most 256 characters; no user content is written to disk. It is protocol behavior evidence, not physical NFC/camera acceptance or a production memory store. The reference implementation's wire tests exercise: discover the card; start a session with modalities; scan an unknown object; send `teach object Workshop router`; verify no stored association yet; send `confirm object`; rescan the same reference and receive the label. A different reference remains unknown. `cancel teaching` cancels a proposal. A new scanned object invalidates it. Requests `scan an object` and `take a photo` exercise collection messages, correlation, wrong-modality rejection, and single consumption. Actual hardware collection requires separate device acceptance. The public golden trace is a synthetic protocol example; its schema/order checks alone cannot prove an agent's memory or confirmation behavior. The independent reference-agent wire tests provide that behavioral evidence. Implementers must test these outcomes against their own agents, without copying the reference's storage/model design. ## Public artifacts All relative paths below resolve from https://agentcall.smartycode.com/ . The schemas are a generated public projection of the canonical contracts, with only the required dependency closure included. Schema omissions do not publish or authorize other private extensions. - `schemas/agentcall-card-0.1.schema.json`: public discovery ingest profile. - `schemas/agentcall-agui-ws-client-0.1.schema.json`: client input messages. - `schemas/agentcall-events-0.1.schema.json`: normalized server envelopes. - `schemas/agentcall-physical-observation-0.1.schema.json`: object/photo input. - `schemas/agentcall-object-passport-0.1.schema.json`: portable reference data. - `protocol/0.1/physical-conformance.json`: positive/negative extension fixtures. - `protocol/0.1/manifest.json`: release version, canonical source commit, upstream pins, exact public file inventory, and SHA-256 hashes. Use the official A2A conformance tooling for the base Agent Card. The AgentCall fixtures additionally check URI migration, modality/profile gates, payload integrity, and the documented physical interaction sequence.