Skip to main content
The Toolkit works with zero configuration. Everything below is optional.

​
Labels

Attach labels at enrollment to group hosts by environment, region, or team. Labels flow through to every question your assistant asks.

​
Collectors

Host facts are durable identity and placement attributes attached to each host in your graph. They size the host (CPU count, total memory) and, on a cloud instance, add a region attribute so co-location and blast-radius queries work across substrates. Region detection is a best-effort probe of the instance metadata service that runs once per session and is a silent no-op on bare metal. The process inventory collector is the “what’s actually running here” view for bare-metal and non-systemd hosts: containers with a bare init, *BSD, macOS, Windows, sysvinit, and openrc. It reads the process table directly and projects each long-running process onto a service node in your graph. Questions like “is postgres up on this box?” work even where the Services collector has nothing to report. To keep the graph readable, the collector only inventories daemon-like processes. It admits a process when the init process is its parent, or when it has been running for at least ten minutes. Transient shells, cron one-shots, and build subprocesses never make it in. Processes that share a name collapse onto a single node — for example, an nginx master and its workers, or a postgres backend pool. The longest-running instance represents the group, so attributes don’t churn as short-lived siblings come and go. On a systemd host, the unit and the process are tracked as separate nodes. The Services collector owns the unit’s lifecycle (active state, restarts), and the process inventory owns the live runtime attributes (pid, cmdline, user, start time). They complement each other; neither overwrites the other.

​
Proxy egress screen

When your assistant uses the Toolkit’s HTTP proxy to reach a cloud-supplied URL, the request is dialed from inside your network. The egress screen stops that path from being turned into an SSRF pivot at cloud instance-metadata or other sensitive addresses. The screen runs post-DNS against the resolved IP, so a hostname that resolves into a denied range is caught too. The screen has three layers:
  • Unconditional floor — 169.254.0.0/16 (IPv4 link-local, including the AWS, GCP, and Azure IMDS address 169.254.169.254), fe80::/10 (IPv6 link-local), and fd00:ec2::/32 (AWS IPv6 IMDS) are always denied. This cannot be disabled.
  • proxy.block-cidrs — additional denied ranges layered on top of the floor. Use this to block an internal admin subnet or loopback that the proxy should never reach.
  • proxy.allow-cidrs — when non-empty, switches the screen into a strict allow-list: a target IP must fall inside one of these ranges or the dial is refused. RFC1918 and loopback are not denied by default — the proxy is meant to reach internal services — so strict egress is opt-in here.
Deny always wins over allow. An IP in any blocked range is refused even if it also matches proxy.allow-cidrs. A malformed CIDR fails the Toolkit at startup rather than silently degrading the screen.

​
Example

​
Environment variables

The same knobs are available as comma-separated environment variables for back-compat with earlier releases:
Leaving both empty (the default) keeps the metadata floor on with RFC1918 and loopback reachable.

​
Service metadata

When the Toolkit runs as the cluster controller, it enriches each Service it emits with metadata read from the workload’s labels and annotations. Labels are preferred over annotations, and ao.-prefixed keys are checked first so an explicit AO key always wins over an incidental application label. Values are matched case-insensitively against common synonyms — for example production, prd, and live all normalize to prod. If no env label is set, the controller falls back to scanning the namespace name (so checkout-prod still resolves to prod). Unrecognized values are omitted rather than guessed. image (the first container’s image) and replicas_desired (the spec replica count, omitted for DaemonSets) come straight from the workload spec and need no configuration. Override the probed keys when your fleet uses different conventions:
Keys are tried in order; the first non-empty value wins. Leave a list empty to keep the built-in defaults.

​
Capabilities

Capabilities control whether the Toolkit on a given host will run a sensitive verb at all — actions like running a shell command, patching a Kubernetes manifest, draining a node, or writing through the HTTP proxy. Read-only verbs (host info, package lists, metric queries, read-only proxy requests) are never gated by this control. Two independent gates decide whether a sensitive verb runs:
  • AO Cloud IAM decides who may invoke a verb against which Toolkit. Manage this in AO Cloud under IAM (policies, roles, action sets, resource sets).
  • On-box capabilities.allow / capabilities.deny decides what this host will run for anyone. Set in the Toolkit’s config file (or via environment variables) on the host itself.
These two gates are independent. A verb runs only if IAM allows the caller and the on-box policy allows the host. The on-box gate is the un-forgeable floor: it lives on the host and a compromised control plane cannot turn it off remotely.

​
Default posture

With no on-box config, the Toolkit is permissive on the box and relies on IAM to gate who can invoke a verb. Tighten an individual host with capabilities.deny, or pin it to a fixed set with capabilities.allow.
This includes the Kubernetes cluster controller: its shipped StatefulSet manifest is permissive at the capability layer too, so a fresh install relies on IAM to gate destructive verbs. Because the controller can act on the whole cluster, it’s a good candidate for on-box hardening — see Harden the cluster controller.

​
How the on-box gate evaluates

For each sensitive verb:
  • deny — always wins. A verb listed here is off, regardless of IAM or allow.
  • allow empty (default) — permissive: the host will run the verb if IAM authorizes the caller.
  • allow non-empty — authoritative whitelist: the host runs exactly the verbs matching allow (minus anything in deny) and nothing else. This is robust against new sensitive verbs added in future Toolkit releases — they stay off until you list them.

​
Configure on-box rules

Entries are globs over the full <plugin>.<verb> name: exact (shell.exec), a whole family (proxy.*), or everything (*). Pin this host to a fixed set:
Or stay permissive and just turn one verb off:
Equivalent environment variables (comma-separated):
A malformed glob in the on-box config fails the Toolkit at boot so you catch the typo immediately.

​
Harden the cluster controller

The Kubernetes cluster controller ships permissive: with no on-box capability config it relies on IAM, same as any other host. Because one controller pod can act across the whole cluster, you may want to add an on-box floor so destructive verbs can’t run there even if IAM would allow them. The reference manifest ships the env hook commented out for exactly this — set AO_CAPABILITIES_DENY (or AO_CAPABILITIES_ALLOW) on the k8s-controller container. A good baseline is to drop arbitrary shell exec — the controller never needs an interactive shell — while leaving the mutating Kubernetes verbs IAM-gated:
To lock it down further, switch to an authoritative allowlist so only the verbs you name can run (everything else stays off, including sensitive verbs added in future Toolkit releases):
deny always wins over IAM and over allow, and these gates use the same glob semantics described above. The Toolkit binary’s own default is unchanged and stays permissive, so Linux hosts and the per-node DaemonSet are unaffected.

​
Kubernetes observation scope

The cluster controller watches every namespace by default. Scope it to a subset with k8s.namespaces-allow (allowlist) or k8s.namespaces-deny (blocklist). Both empty watches everything; the two lists are mutually exclusive.
Scope is enforced at the apiserver for watches and on every verb the controller can run:
  • Observation — the controller does not list or watch resources outside scope, so they never appear in the cluster graph.
  • Actions — scale.set, rollout.restart, rollout.undo, and rollout.status against a workload in an out-of-scope namespace are refused with a clear “outside this controller’s configured observation scope” error.
  • Events — events.list skips out-of-scope namespaces. A pinned namespace argument outside scope is refused; a cluster-wide list drops out-of-scope events client-side. Cluster-scoped events (for example, Node events with no involved-object namespace) always pass through.
This is a hard boundary, not a default. If you scope a controller out of kube-system, nothing routed through it — read or write — can touch kube-system. Run a second controller with a different scope if you need split coverage with different IAM around each.