Send events from applications on a host

Applications that run on a host with the Toolkit can send events to AO through the Toolkit, without an HMAC secret and without knowing any AO identifiers. Examples include a health checker opening and closing incidents, a configuration-management run reporting its result, or a deploy job announcing a rollout. The Toolkit is already enrolled, already queues and retries, and already knows which host it is on. It accepts the event locally, attaches it to the host, and delivers it to AO. There are three local entry points. All of them use the same ao.webhook.v1 records as the AO Webhook:

Entry point	Best for	Works while the Toolkit is down?
Unix socket	long-running services and containers	no: retry, or fall back to the spool
Spool directory	cron jobs, scripts, configuration-management states	yes: picked up on start
`ao-toolkit emit`	shell scripts	yes: falls back to the spool

For systems that don’t run on a Toolkit host, such as a SaaS CI service, use the AO Webhook instead. Records are identical, so an emitter can switch transports without changing them.

The local socket

The Toolkit listens on a Unix stream socket and speaks plain HTTP/1.1, so curl and every language’s standard HTTP client can use it:

curl --unix-socket /run/ao-toolkit/events.sock \
  -H 'Content-Type: application/json' \
  -d @event.json \
  http://local/v1/events

The host name in the URL is ignored. Any value works.

Setting	Default	Environment variable
Socket path	`/run/ao-toolkit/events.sock`	`AO_EVENTS_SOCKET` (empty turns the socket off)
Socket group	the Toolkit’s group	`AO_EVENTS_SOCKET_GROUP`

The socket is created with mode 0660. Whether a process may connect is an ordinary file-permission decision: add the emitting service’s user to the socket’s group, or run the emitter as root. The Toolkit never opens a TCP listener for local events. A loopback port can’t tell callers apart, and every local process can reach it. A minimal event body:

event.json

{
  "schema": "ao.webhook.v1",
  "records": [
    {
      "type": "event",
      "source_event_id": "healthcheck-rpc-2026-10-21T10:00:00Z",
      "kind": "acme:incident",
      "opened_at": "2026-10-21T10:00:00Z",
      "title": "RPC endpoint not responding",
      "tags": { "severity": "warning" }
    }
  ]
}

With no affected_entities, the event attaches to the Toolkit’s own host. A successful request returns 200:

{ "accepted": 1, "host_id": "847cdd30-…", "source": "healthcheck.service" }

source is how the Toolkit identified your process (see Who sent it). The Toolkit answers only after the event is written to its on-disk delivery queue. An acknowledged event survives a Toolkit restart or upgrade and is delivered once the Toolkit reconnects.

Toolkit in a container

When the Toolkit itself runs in a container, two things are required:

The host PID namespace (--pid host, pid: host, or hostPID: true). The Toolkit identifies callers by their process ID. Without the host PID namespace it cannot see processes in other containers and rejects them with unidentified_peer (or labels them only by user ID, as your policy allows).
A shared socket directory. Bind-mount the host’s /run/ao-toolkit into the Toolkit container, and the same directory into every emitter container.

# Toolkit (add to your existing Toolkit flags)
docker run -d --name ao-toolkit --pid host \
  -v /run/ao-toolkit:/run/ao-toolkit \
  …

# An emitting container
docker run -d --name healthcheck \
  -v /run/ao-toolkit:/run/ao-toolkit \
  my-healthcheck:latest

services:
  ao-toolkit:
    # … your existing Toolkit service …
    pid: host
    volumes:
      - /run/ao-toolkit:/run/ao-toolkit

  healthcheck:
    image: my-healthcheck:latest
    volumes:
      - /run/ao-toolkit:/run/ao-toolkit

# Toolkit DaemonSet pod spec (add to the existing spec)
spec:
  hostPID: true
  containers:
    - name: ao-toolkit
      volumeMounts:
        - name: ao-events
          mountPath: /run/ao-toolkit
  volumes:
    - name: ao-events
      hostPath:
        path: /run/ao-toolkit
        type: DirectoryOrCreate
---
# Emitting pod on the same node
spec:
  containers:
    - name: healthcheck
      volumeMounts:
        - name: ao-events
          mountPath: /run/ao-toolkit
  volumes:
    - name: ao-events
      hostPath:
        path: /run/ao-toolkit
        type: Directory

The emitter container’s user must be allowed to open the socket (mode 0660): run it as root, or give it the socket group’s GID.

Who sent it

On every connection the Toolkit reads the caller’s process ID and user ID from the kernel and resolves them to the systemd unit or container the process belongs to. It records that on every event as tags the emitter cannot override:

Tag	Value
`ao.source`	the source label: the unit name, `container:<name>`, or `uid:<n>`
`ao.source-unit`	the systemd unit, when the caller runs in one
`ao.source-container`	the container, when the caller runs in one
`ao.source-uid`	the caller’s user ID

Events are also stored with that source, so events from different services never overwrite each other, even if they use the same source_event_id.

Policy: who may send what

Without a policy file, any process that can open the socket may send any namespaced event kind about its own host and that host’s services. To restrict sources, point AO_EVENTS_POLICY at a YAML file. Once a policy file exists, everything it does not allow is denied:

/etc/ao-toolkit/events-policy.yaml

sources:
  - match: { unit: salt-minion.service }
    kinds: ["salt:*"]
  - match: { container: healthcheck }
    kinds: ["acme:*"]
    entities: [own_host, own_services]
  - match: { unit: deployd.service }
    kinds: ["deploy:*"]
    entities: [any_host]        # an orchestrator reporting on other hosts
  - match: { uid: 0 }
    kinds: ["backup:*"]

Field	Meaning
`match`	one of `unit`, `container`, or `uid`
`kinds`	event kinds this source may send; `*` is a wildcard
`entities`	which hosts and services the source may reference: `own_host` (the default), `own_services`, `any_host`

A request with a kind outside the source’s list is rejected with forbidden_kind, and a reference outside its entities with forbidden_entity. Nothing from a rejected request is stored.

Attach events to the host and its services

You never need AO identifiers. Use self and the Toolkit fills in its own host:

"affected_entities": [
  { "kind": "Host", "identity_tags": { "host_id": "self" } },
  { "kind": "HostService", "identity_tags": {
      "host_id": "self", "name": "nginx.service", "service_manager": "systemd" } }
]

Entity kinds (Host, HostService, Service) are matched case-insensitively and stored in that spelling.

Events about other hosts

An orchestrator such as a configuration-management master or a deploy runner reports on other machines that it knows by name. A source whose policy allows any_host can reference a host by any one of these keys:

{ "kind": "Host", "identity_tags": { "hostname": "web-03" } }

Key	Matches
`hostname`	the host’s name
`fqdn`	its fully qualified domain name
`instance_name`	its cloud instance name
`toolkit_id`	the ID of the Toolkit enrolled on it

AO resolves the reference to the one host in your organization with that value and stores the canonical host reference. A reference that matches no host, or several, is kept as sent and listed as unresolved, so you can see it instead of the event silently attaching to nothing.

Updates, closing, and retries

Events are identified by source_event_id within a source on a host. Sending the same ID again updates the event, which makes retries safe:

To close an event, send the same source_event_id with closed_at set.
AO keeps the version with the latest observed_at, so set observed_at to the time of each update.

The socket exists only while the Toolkit runs. If it is missing (for example during an upgrade), retry with backoff, or write to the spool.

The spool directory

For cron jobs and scripts, drop a file instead of making a request:

Setting	Default	Environment variable
Spool directory	`/var/lib/ao-toolkit/spool/`	`AO_EVENTS_SPOOL`

Write the ao.webhook.v1 body to <anything>.json.tmp in the spool directory.
Rename it to <anything>.json. Renaming is atomic, so the Toolkit never reads a half-written file.

The Toolkit picks up *.json files, validates them, queues the events, and deletes the files. A file that fails validation moves to spool/rejected/ with a matching .error file explaining why. Files written while the Toolkit is down are processed when it starts. Spooled events are attributed to the file’s owner (uid:<n>). Policy rules can match on uid.

`ao-toolkit emit`

For shell scripts and configuration-management states, the Toolkit binary sends an event for you. It uses the socket and falls back to the spool when the Toolkit isn’t running:

ao-toolkit emit \
  --kind salt:highstate \
  --id "highstate-$(date +%s)" \
  --title "Highstate applied" \
  --tag severity=info --tag env=production \
  --payload @/tmp/highstate-summary.json \
  --entity Host:self

# close an open event
ao-toolkit emit --kind acme:maintenance --id maint-42 --closed

--entity HostService:<unit> attaches the event to one of the host’s systemd services.

Limits

Each source has its own rate limit (default 50 events per second with a burst of 500, configurable with AO_EVENTS_RATE and AO_EVENTS_BURST). A source over its limit receives rate_limited and should back off. One noisy source never delays other sources or the Toolkit’s own monitoring.

Check the Toolkit

ao-toolkit status shows the host ID, the socket and spool paths, and per-source accepted and rejected counts.

Verify with your AI assistant

If your assistant is connected to the AO MCP server, it can do the whole integration from there. These are discovery tools: ask the assistant to call find_tools if it doesn’t see them.

integration_guide: step-by-step instructions for your task, for example “emit events from a containerized app on a host”.
host_identity: the host’s ID and the exact affected_entities references for it and its services.
validate_event: a dry run that checks your records and resolves every reference against your graph, for example “attaches to Host 847cdd30-…” or “orphaned: no Service with name=checkout”. Nothing is stored.
Send the event.
event_search with an affectedSelector on the host finds it.

This documentation is also available to the assistant through the docs_search and docs_get tools.

Reference

Event record (`ao.webhook.v1`)

Envelope: {"schema": "ao.webhook.v1", "records": [ … ]}. Ingested record types: event. Recognized but not ingested yet (unsupported_type): graph.attribute_affirm, graph.attribute_remove, graph.edge, graph.edge_remove, graph.node, graph.node_remove, telemetry.

Field	Type	Required	Description
`type`	string	yes	Record type. Always “event”.
`source_event_id`	string	yes	Your stable id for this event. Re-sending the same id updates the event (send closed_at to close it).
`kind`	string	yes	Namespaced event kind, e.g. deploy:started. Lowercase; namespace and subkind joined by : or .
`title`	string	no	Short human-readable summary.
`opened_at`	string (RFC 3339)	yes	When the event started (RFC 3339).
`closed_at`	string (RFC 3339)	no	When it ended (not before opened_at). Equal to opened_at for a point-in-time event; omit while it is still open.
`observed_at`	string (RFC 3339)	no	When you observed this state. Defaults to opened_at; must move forward on every update of the same source_event_id.
`affected_entities`	array	no	Graph nodes the event is about: [{kind, identity_tags}], e.g. {“kind”:“Host”,“identity_tags”:{“host_id”:”…”}}. The kind is a graph kind (Host, HostService, Service, …), matched case-insensitively; identity values are canonicalized.
`tags`	object (string → string)	no	Flat string map of filter dimensions. tags.severity sets the event severity (critical, warning, info, or a synonym).
`payload`	object	no	Free-form JSON object with the event body.

Each affected_entities entry:

Field	Type	Required	Description
`kind`	string	yes	Graph kind: BatchJob, Cluster, Host, HostScheduledTask, HostService, HttpEndpoint, Region, Replica, Repository, ScheduledTask, Service, Team. Matched case-insensitively against these and stored in this spelling.
`identity_tags`	object	yes	The node’s identity: Host {host_id}, HostService {host_id, name, service_manager}, Service {name, …}. Values are canonicalized (lowercase, - _ . and spaces collapse to -).

kind must match ^[a-z][a-z0-9_-]*[:.][a-z][a-z0-9_-]+$.

Severity

Set tags.severity. These labels are recognized (case-insensitive); anything else leaves the event without a severity.

Severity	Accepted labels
`critical`	`critical`, `crit`, `fatal`, `emergency`, `emerg`, `alert`, `page`, `sev1`, `sev-1`, `p1`, `error`, `err`
`warning`	`warning`, `warn`, `minor`, `sev2`, `sev-2`, `p2`, `degraded`
`info`	`info`, `informational`, `notice`, `ok`, `resolved`, `low`, `sev3`, `sev-3`, `p3`, `p4`, `p5`

Limits

Limit	Value
Records per request	500
Request body	1 MiB
One record	64 KiB
Local rate per source	50 events/s, burst 500 (`AO_EVENTS_RATE`, `AO_EVENTS_BURST`)

Error codes: AO Webhook

Errors answer {"error": {"code": "…", "message": "…"}}.

Code	HTTP	Meaning
`invalid_schema`	400	The envelope or a record is structurally invalid (bad schema string, no records, a missing required field, a malformed kind, an unknown record type, a payload that is not an object). Fix the request; do not retry it unchanged.
`missing_signature`	401	X-AO-Timestamp or X-AO-Signature-256 is absent or malformed.
`bad_signature`	401	The HMAC did not verify against the install’s secret.
`expired_timestamp`	401	X-AO-Timestamp is outside the ±300 s replay window (check the sender’s clock).
`revoked`	403	The install exists but is not active.
`not_found`	404	No AO Webhook install matches the id in the URL.
`payload_too_large`	413	The body exceeds 1 MiB, or one record exceeds 64 KiB. Split the batch.
`unsupported_type`	422	A recognized ao.webhook.v1 record type that is not ingested yet (graph.*, telemetry). The whole batch is refused.
`type_not_accepted`	422	The record type is not in this install’s accepted types.
`internal_error`	500	An ingest-side failure. Retry with backoff; re-sending the same source_event_id is safe.

Error codes: toolkit socket

Errors answer {"error": {"code": "…", "message": "…"}}.

Code	HTTP	Meaning
`invalid_schema`	400	The envelope or a record is structurally invalid (bad schema string, no records, a missing required field, a malformed kind, an unknown record type, a payload that is not an object). Fix the request; do not retry it unchanged.
`forbidden_kind`	403	The policy does not allow this source to send this kind. Nothing from the request is stored.
`forbidden_entity`	403	A record references an entity the source’s policy does not allow (another host without any_host, another host’s service). Nothing is stored.
`unidentified_peer`	403	The toolkit could not identify the caller (no host pid namespace, or the process exited) and the policy needs its identity.
`payload_too_large`	413	The body exceeds 1 MiB, or one record exceeds 64 KiB. Split the batch.
`unsupported_type`	422	A recognized ao.webhook.v1 record type that is not ingested yet (graph.*, telemetry). The whole batch is refused.
`rate_limited`	429	This source is over its rate limit. Back off and retry; other sources and the toolkit’s own collectors are unaffected.
`internal_error`	500	An ingest-side failure. Retry with backoff; re-sending the same source_event_id is safe.
`queue_full`	503	The toolkit’s disk queue is full (it cannot reach AO). Retry later, or write to the spool.

Host references

Reference	Meaning
`{"kind":"Host","identity_tags":{"host_id":"self"}}`	The receiving host. The toolkit fills in its host_id.
`{"kind":"HostService","identity_tags":{"host_id":"self","name":"<unit>","service_manager":"systemd"}}`	A systemd service on the receiving host.
`(no affected_entities)`	Local intake only: the event attaches to the receiving host.

Cross-host keys (Local sources whose policy allows any_host. On the webhook, reference a host by host_id (host_identity returns it).): hostname, fqdn, instance_name, toolkit_id. Resolved at ingest to the one host with that value in the organization and stored as {“kind”:“Host”,“identity_tags”:{“host_id”:…}}. Zero or several matches: stored as sent and listed by integration_unresolved_refs (reason no_match / ambiguous).

Provenance tags

The toolkit sets these on every local event:

Tag	Meaning
`ao.source`	The local source: a systemd unit (salt-minion.service), container:<name>, or uid:<n>.
`ao.source-unit`	The emitting process’s systemd unit, when it has one.
`ao.source-container`	The emitting process’s container name, when it runs in one.
`ao.source-pod`	The emitting process’s Kubernetes pod, when it runs in one.
`ao.source-uid`	The emitting process’s uid (the file owner for a spooled file).
`ao.host-id`	The host_id of the host whose toolkit received the event.
`ao.hostname`	That host’s hostname.
`ao.toolkit-id`	That toolkit’s id.

Getting started

Toolkit

Integrations

MCP

Send events from applications on a host

The local socket

Toolkit in a container

Who sent it

Policy: who may send what

Attach events to the host and its services

Events about other hosts

Updates, closing, and retries

The spool directory

`ao-toolkit emit`

Limits

Check the Toolkit

Verify with your AI assistant

Reference

Event record (`ao.webhook.v1`)

Severity

Limits

Error codes: AO Webhook

Error codes: toolkit socket

Host references

Provenance tags

​The local socket

​Toolkit in a container

​Who sent it

​Policy: who may send what

​Attach events to the host and its services

​Events about other hosts

​Updates, closing, and retries

​The spool directory

​ao-toolkit emit

​Limits

​Check the Toolkit

​Verify with your AI assistant

​Reference

​Event record (ao.webhook.v1)

​Severity

​Limits

​Error codes: AO Webhook

​Error codes: toolkit socket

​Host references

​Provenance tags

The local socket

Toolkit in a container

Who sent it

Policy: who may send what

Attach events to the host and its services

Events about other hosts

Updates, closing, and retries

The spool directory

`ao-toolkit emit`

Limits

Check the Toolkit

Verify with your AI assistant

Reference

Event record (`ao.webhook.v1`)

Severity

Limits

Error codes: AO Webhook

Error codes: toolkit socket

Host references

Provenance tags