> ## Documentation Index
> Fetch the complete documentation index at: https://docs.automatedoperations.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Send events from applications on a host

> Send events from processes, containers, cron jobs, and scripts on a host through the Toolkit's local socket, with no secret to manage. The Toolkit attaches each event to its host and delivers it.

Applications that run on a host with the Toolkit can send events to AO
**through the Toolkit**, without an HMAC secret and without knowing any AO
identifiers. Examples include a health checker opening and closing
incidents, a configuration-management run reporting its result, or a
deploy job announcing a rollout. The Toolkit is already enrolled, already
queues and retries, and already knows which host it is on. It accepts the
event locally, attaches it to the host, and delivers it to AO.

There are three local entry points. All of them use the same
[`ao.webhook.v1`](/integrations/ao-webhooks#the-request-body) records as
the AO Webhook:

| Entry point | Best for | Works while the Toolkit is down? |
| - | - | - |
| [Unix socket](#the-local-socket) | long-running services and containers | no: retry, or fall back to the spool |
| [Spool directory](#the-spool-directory) | cron jobs, scripts, configuration-management states | yes: picked up on start |
| [`ao-toolkit emit`](#ao-toolkit-emit) | shell scripts | yes: falls back to the spool |

For systems that don't run on a Toolkit host, such as a SaaS CI service,
use the [AO Webhook](/integrations/ao-webhooks.md) instead. Records are
identical, so an emitter can switch transports without changing them.

## The local socket

The Toolkit listens on a Unix stream socket and speaks plain HTTP/1.1, so
`curl` and every language's standard HTTP client can use it:

```bash theme={null}
curl --unix-socket /run/ao-toolkit/events.sock \
  -H 'Content-Type: application/json' \
  -d @event.json \
  http://local/v1/events
```

The host name in the URL is ignored. Any value works.

| Setting | Default | Environment variable |
| - | - | - |
| Socket path | `/run/ao-toolkit/events.sock` | `AO_EVENTS_SOCKET` (empty turns the socket off) |
| Socket group | the Toolkit's group | `AO_EVENTS_SOCKET_GROUP` |

The socket is created with mode `0660`. Whether a process may connect is
an ordinary file-permission decision: add the emitting service's user to
the socket's group, or run the emitter as root. The Toolkit never opens a
TCP listener for local events. A loopback port can't tell callers apart,
and every local process can reach it.

A minimal event body:

```json event.json theme={null}
{
  "schema": "ao.webhook.v1",
  "records": [
    {
      "type": "event",
      "source_event_id": "healthcheck-rpc-2026-10-21T10:00:00Z",
      "kind": "acme:incident",
      "opened_at": "2026-10-21T10:00:00Z",
      "title": "RPC endpoint not responding",
      "tags": { "severity": "warning" }
    }
  ]
}
```

With no `affected_entities`, the event attaches to the Toolkit's own host.
A successful request returns `200`:

```json theme={null}
{ "accepted": 1, "host_id": "847cdd30-…", "source": "healthcheck.service" }
```

`source` is how the Toolkit identified your process (see
[Who sent it](#who-sent-it)).

The Toolkit answers only after the event is written to its on-disk
delivery queue. An acknowledged event survives a Toolkit restart or
upgrade and is delivered once the Toolkit reconnects.

### Toolkit in a container

When the Toolkit itself runs in a container, three things are required:

1. **The host PID namespace** (`--pid host`, `pid: host`, or `hostPID: true`).
   The Toolkit identifies callers by their process ID. Without the host
   PID namespace it cannot see processes in other containers and rejects
   them with `unidentified_peer` (or labels them only by user ID, as your
   policy allows).
2. **A shared socket directory.** Bind-mount the host's `/run/ao-toolkit`
   into the Toolkit container, and the same directory into every emitter
   container.
3. **Container names.** The Toolkit names a calling container from the
   runtime's files on the host, read under `AO_HOST_ROOT` (for example
   `/host`): `/var/lib/docker/containers` on Docker, `/var/log/containers`
   on Kubernetes. Mount it read-only. Without it a container is known only
   by its 12-character ID, so `ao.source-container` shows the ID and a
   policy rule such as `match: {container: healthcheck}` does not match.

```bash docker run theme={null}
# Toolkit (add to your existing Toolkit flags)
docker run -d --name ao-toolkit --pid host \
  -v /run/ao-toolkit:/run/ao-toolkit \
  -v /var/lib/docker/containers:/host/var/lib/docker/containers:ro \
  -e AO_HOST_ROOT=/host \
  …

# An emitting container
docker run -d --name healthcheck \
  -v /run/ao-toolkit:/run/ao-toolkit \
  my-healthcheck:latest
```

```yaml compose.yaml theme={null}
services:
  ao-toolkit:
    # … your existing Toolkit service …
    pid: host
    environment:
      AO_HOST_ROOT: /host
    volumes:
      - /run/ao-toolkit:/run/ao-toolkit
      - /var/lib/docker/containers:/host/var/lib/docker/containers:ro

  healthcheck:
    image: my-healthcheck:latest
    volumes:
      - /run/ao-toolkit:/run/ao-toolkit
```

```yaml Kubernetes theme={null}
# Toolkit DaemonSet pod spec (add to the existing spec)
spec:
  hostPID: true
  containers:
    - name: ao-toolkit
      env:
        - name: AO_HOST_ROOT
          value: /host
      volumeMounts:
        - name: ao-events
          mountPath: /run/ao-toolkit
        - name: container-logs
          mountPath: /host/var/log/containers
          readOnly: true
  volumes:
    - name: ao-events
      hostPath:
        path: /run/ao-toolkit
        type: DirectoryOrCreate
    - name: container-logs
      hostPath:
        path: /var/log/containers
---
# Emitting pod on the same node
spec:
  containers:
    - name: healthcheck
      volumeMounts:
        - name: ao-events
          mountPath: /run/ao-toolkit
  volumes:
    - name: ao-events
      hostPath:
        path: /run/ao-toolkit
        type: Directory
```

The emitter container's user must be allowed to open the socket (mode
`0660`): run it as root, or give it the socket group's GID.

## Who sent it

On every connection the Toolkit reads the caller's process ID and user ID
from the kernel and resolves them to the **systemd unit** or **container**
the process belongs to. It records that on every event as tags the
emitter cannot override:

| Tag | Value |
| - | - |
| `ao.source` | the source label: the unit name, `container:<name>`, or `uid:<n>` |
| `ao.source-unit` | the systemd unit, when the caller runs in one |
| `ao.source-container` | the container, when the caller runs in one |
| `ao.source-uid` | the caller's user ID |

Events are also stored with that source, so events from different
services never overwrite each other, even if they use the same
`source_event_id`.

## Policy: who may send what

Without a policy file, any process that can open the socket may send any
namespaced event kind about its own host and that host's services.

To restrict sources, point `AO_EVENTS_POLICY` at a YAML file. Once a
policy file exists, **everything it does not allow is denied**:

```yaml /etc/ao-toolkit/events-policy.yaml theme={null}
sources:
  - match: { unit: salt-minion.service }
    kinds: ["salt:*"]
  - match: { container: healthcheck }
    kinds: ["acme:*"]
    entities: [own_host, own_services]
  - match: { unit: deployd.service }
    kinds: ["deploy:*"]
    entities: [any_host]        # an orchestrator reporting on other hosts
  - match: { uid: 0 }
    kinds: ["backup:*"]
```

| Field | Meaning |
| - | - |
| `match` | one of `unit`, `container`, or `uid` |
| `kinds` | event kinds this source may send; `*` is a wildcard |
| `entities` | which hosts and services the source may reference: `own_host` (the default), `own_services`, `any_host` |

A request with a kind outside the source's list is rejected with
`forbidden_kind`, and a reference outside its `entities` with
`forbidden_entity`. Nothing from a rejected request is stored.

## Attach events to the host and its services

You never need AO identifiers. Use `self` and the Toolkit fills in its
own host:

```json theme={null}
"affected_entities": [
  { "kind": "Host", "identity_tags": { "host_id": "self" } },
  { "kind": "HostService", "identity_tags": {
      "host_id": "self", "name": "nginx.service", "service_manager": "systemd" } }
]
```

Entity kinds (`Host`, `HostService`, `Service`) are matched case-insensitively and stored in that spelling.

### Events about other hosts

An orchestrator such as a configuration-management master or a deploy
runner reports on other machines that it knows by name. A source whose
policy allows `any_host` can reference a host by any one of these keys:

```json theme={null}
{ "kind": "Host", "identity_tags": { "hostname": "web-03" } }
```

| Key | Matches |
| - | - |
| `hostname` | the host's name |
| `fqdn` | its fully qualified domain name |
| `instance_name` | its cloud instance name |
| `toolkit_id` | the ID of the Toolkit enrolled on it |

AO resolves the reference to the one host in your organization with that
value and stores the canonical host reference. A reference that matches
no host, or several, is kept as sent and listed as unresolved, so you can
see it instead of the event silently attaching to nothing.

## Updates, closing, and retries

Events are identified by `source_event_id` within a source on a host.
Sending the same ID again updates the event, which makes retries safe:

* To close an event, send the same `source_event_id` with `closed_at` set.
* AO keeps the version with the latest `observed_at`, so set
  `observed_at` to the time of each update.

The socket exists only while the Toolkit runs. If it is missing (for
example during an upgrade), retry with backoff, or write to the spool.

## The spool directory

For cron jobs and scripts, drop a file instead of making a request:

| Setting | Default | Environment variable |
| - | - | - |
| Spool directory | `/var/lib/ao-toolkit/spool/` | `AO_EVENTS_SPOOL` |

1. Write the `ao.webhook.v1` body to `<anything>.json.tmp` in the spool
   directory.
2. Rename it to `<anything>.json`. Renaming is atomic, so the Toolkit
   never reads a half-written file.

The Toolkit picks up `*.json` files, validates them, queues the events,
and deletes the files. A file that fails validation moves to
`spool/rejected/` with a matching `.error` file explaining why. Files
written while the Toolkit is down are processed when it starts.

Spooled events are attributed to the file's owner (`uid:<n>`). Policy
rules can match on `uid`.

## `ao-toolkit emit`

For shell scripts and configuration-management states, the Toolkit binary
sends an event for you. It uses the socket and falls back to the spool
when the Toolkit isn't running:

```bash theme={null}
ao-toolkit emit \
  --kind salt:highstate \
  --id "highstate-$(date +%s)" \
  --title "Highstate applied" \
  --tag severity=info --tag env=production \
  --payload @/tmp/highstate-summary.json \
  --entity Host:self

# close an open event
ao-toolkit emit --kind acme:maintenance --id maint-42 --closed
```

`--entity HostService:<unit>` attaches the event to one of the host's
systemd services.

## Limits

Each source has its own rate limit (default 50 events per second with a
burst of 500, configurable with `AO_EVENTS_RATE` and `AO_EVENTS_BURST`).
A source over its limit receives `rate_limited` and should back off. One
noisy source never delays other sources or the Toolkit's own monitoring.

## Check the Toolkit

`ao-toolkit status` shows the host ID, the socket and spool paths, and
per-source accepted and rejected counts.

## Verify with your AI assistant

If your assistant is connected to the AO MCP server, it can do the whole
integration from there. These are discovery tools: ask the assistant to
call `find_tools` if it doesn't see them.

1. `integration_guide`: step-by-step instructions for your task, for
   example "emit events from a containerized app on a host".
2. `host_identity`: the host's ID and the exact `affected_entities`
   references for it and its services.
3. `validate_event`: a dry run that checks your records and resolves every
   reference against your graph, for example "attaches to Host
   847cdd30-…" or "orphaned: no Service with name=checkout". Nothing is
   stored.
4. Send the event.
5. `event_search` with an `affectedSelector` on the host finds it.

This documentation is also available to the assistant through the
`docs_search` and `docs_get` tools.

## Reference

### Event record (`ao.webhook.v1`)

Envelope: `{"schema": "ao.webhook.v1", "records": [ … ]}`. Ingested record types: `event`. Recognized but not ingested yet (`unsupported_type`): `graph.attribute_affirm`, `graph.attribute_remove`, `graph.edge`, `graph.edge_remove`, `graph.node`, `graph.node_remove`, `telemetry`.

| Field | Type | Required | Description |
| - | - | - | - |
| `type` | string | yes | Record type. Always "event". |
| `source_event_id` | string | yes | Your stable id for this event. Re-sending the same id updates the event (send closed\_at to close it). |
| `kind` | string | yes | Namespaced event kind, e.g. deploy:started. Lowercase; namespace and subkind joined by : or . |
| `title` | string | no | Short human-readable summary. |
| `opened_at` | string (RFC 3339) | yes | When the event started (RFC 3339). |
| `closed_at` | string (RFC 3339) | no | When it ended (not before opened\_at). Equal to opened\_at for a point-in-time event; omit while it is still open. |
| `observed_at` | string (RFC 3339) | no | When you observed this state. Defaults to opened\_at; must move forward on every update of the same source\_event\_id. |
| `affected_entities` | array | no | Graph nodes the event is about: \[\{kind, identity\_tags}], e.g. \{"kind":"Host","identity\_tags":\{"host\_id":"…"}}. The kind is a graph kind (Host, HostService, Service, …), matched case-insensitively; identity values are canonicalized. |
| `tags` | object (string → string) | no | Flat string map of filter dimensions. tags.severity sets the event severity (critical, warning, info, or a synonym). |
| `payload` | object | no | Free-form JSON object with the event body. |

Each `affected_entities` entry:

| Field | Type | Required | Description |
| - | - | - | - |
| `kind` | string | yes | Graph kind: BatchJob, Cluster, Host, HostScheduledTask, HostService, HttpEndpoint, Region, Replica, Repository, ScheduledTask, Service, Team. Matched case-insensitively against these and stored in this spelling. |
| `identity_tags` | object | yes | The node's identity: Host \{host\_id}, HostService \{host\_id, name, service\_manager}, Service \{name, …}. Values are canonicalized (lowercase, - \_ . and spaces collapse to -). |

`kind` must match `^[a-z][a-z0-9_-]*[:.][a-z][a-z0-9_-]+$`.

### Severity

Set `tags.severity`. These labels are recognized (case-insensitive); anything else leaves the event without a severity.

| Severity | Accepted labels |
| - | - |
| `critical` | `critical`, `crit`, `fatal`, `emergency`, `emerg`, `alert`, `page`, `sev1`, `sev-1`, `p1`, `error`, `err` |
| `warning` | `warning`, `warn`, `minor`, `sev2`, `sev-2`, `p2`, `degraded` |
| `info` | `info`, `informational`, `notice`, `ok`, `resolved`, `low`, `sev3`, `sev-3`, `p3`, `p4`, `p5` |

### Limits

| Limit | Value |
| - | - |
| Records per request | 500 |
| Request body | 1 MiB |
| One record | 64 KiB |
| Local rate per source | 50 events/s, burst 500 (`AO_EVENTS_RATE`, `AO_EVENTS_BURST`) |

### Error codes: AO Webhook

Errors answer `{"error": {"code": "…", "message": "…"}}`.

| Code | HTTP | Meaning |
| - | - | - |
| `invalid_schema` | 400 | The envelope or a record is structurally invalid (bad schema string, no records, a missing required field, a malformed kind, an unknown record type, a payload that is not an object). Fix the request; do not retry it unchanged. |
| `missing_signature` | 401 | X-AO-Timestamp or X-AO-Signature-256 is absent or malformed. |
| `bad_signature` | 401 | The HMAC did not verify against the install's secret. |
| `expired_timestamp` | 401 | X-AO-Timestamp is outside the ±300 s replay window (check the sender's clock). |
| `revoked` | 403 | The install exists but is not active. |
| `not_found` | 404 | No AO Webhook install matches the id in the URL. |
| `payload_too_large` | 413 | The body exceeds 1 MiB, or one record exceeds 64 KiB. Split the batch. |
| `unsupported_type` | 422 | A recognized ao.webhook.v1 record type that is not ingested yet (graph.\*, telemetry). The whole batch is refused. |
| `type_not_accepted` | 422 | The record type is not in this install's accepted types. |
| `internal_error` | 500 | An ingest-side failure. Retry with backoff; re-sending the same source\_event\_id is safe. |

### Error codes: toolkit socket

Errors answer `{"error": {"code": "…", "message": "…"}}`.

| Code | HTTP | Meaning |
| - | - | - |
| `invalid_schema` | 400 | The envelope or a record is structurally invalid (bad schema string, no records, a missing required field, a malformed kind, an unknown record type, a payload that is not an object). Fix the request; do not retry it unchanged. |
| `forbidden_kind` | 403 | The policy does not allow this source to send this kind. Nothing from the request is stored. |
| `forbidden_entity` | 403 | A record references an entity the source's policy does not allow (another host without any\_host, another host's service). Nothing is stored. |
| `unidentified_peer` | 403 | The toolkit could not identify the caller (no host pid namespace, or the process exited) and the policy needs its identity. |
| `payload_too_large` | 413 | The body exceeds 1 MiB, or one record exceeds 64 KiB. Split the batch. |
| `unsupported_type` | 422 | A recognized ao.webhook.v1 record type that is not ingested yet (graph.\*, telemetry). The whole batch is refused. |
| `rate_limited` | 429 | This source is over its rate limit. Back off and retry; other sources and the toolkit's own collectors are unaffected. |
| `internal_error` | 500 | An ingest-side failure. Retry with backoff; re-sending the same source\_event\_id is safe. |
| `queue_full` | 503 | The toolkit's disk queue is full (it cannot reach AO). Retry later, or write to the spool. |

### Host references

| Reference | Meaning |
| - | - |
| `{"kind":"Host","identity_tags":{"host_id":"self"}}` | The receiving host. The toolkit fills in its host\_id. |
| `{"kind":"HostService","identity_tags":{"host_id":"self","name":"<unit>","service_manager":"systemd"}}` | A systemd service on the receiving host. |
| `(no affected_entities)` | Local intake only: the event attaches to the receiving host. |

Cross-host keys (Local sources whose policy allows any\_host. On the webhook, reference a host by host\_id (host\_identity returns it).): `hostname`, `fqdn`, `instance_name`, `toolkit_id`. Resolved at ingest to the one host with that value in the organization and stored as \{"kind":"Host","identity\_tags":\{"host\_id":…}}. Zero or several matches: stored as sent and listed by integration\_unresolved\_refs (reason no\_match / ambiguous).

### Provenance tags

The toolkit sets these on every local event:

| Tag | Meaning |
| - | - |
| `ao.source` | The local source: a systemd unit (salt-minion.service), container:\<name>, or uid:\<n>. |
| `ao.source-unit` | The emitting process's systemd unit, when it has one. |
| `ao.source-container` | The emitting process's container name, when it runs in one. |
| `ao.source-pod` | The emitting process's Kubernetes pod, when it runs in one. |
| `ao.source-uid` | The emitting process's uid (the file owner for a spooled file). |
| `ao.host-id` | The host\_id of the host whose toolkit received the event. |
| `ao.hostname` | That host's hostname. |
| `ao.toolkit-id` | That toolkit's id. |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.