> ## Documentation Index
> Fetch the complete documentation index at: https://docs.automatedoperations.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Configure the Toolkit

> Configure Toolkit labels, collectors, and the on-box capability gate that decides which sensitive verbs each host will run for AO Cloud.

The Toolkit works with zero configuration. Everything below is optional.

## Labels

Attach labels at enrollment to group hosts by environment, region, or
team. Labels flow through to every question your assistant asks.

## Collectors

| Collector         | What it observes                                                                                                                                                                    | Default |
| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- |
| Services          | systemd units: state, restarts, failures                                                                                                                                            | on      |
| Process inventory | long-running processes: pid, cmdline, user, start time                                                                                                                              | on      |
| System metrics    | CPU, memory, disk, network                                                                                                                                                          | on      |
| Host facts        | OS, kernel, architecture, CPU count, memory, FQDN, IP addresses, virtualization, and cloud region (AWS, GCP, Azure) when running on a cloud instance                                | on      |
| Scheduled tasks   | classic cron entries (`/etc/crontab`, `/etc/cron.d`, the run-parts directories, per-user spool crontabs) and systemd `.timer` units, each with its schedule expression (Linux only) | on      |
| Topology          | how services and hosts relate                                                                                                                                                       | on      |

Host facts are durable identity and placement attributes attached to each
host in your graph. They size the host (CPU count, total memory) and,
on a cloud instance, add a region attribute so co-location and
blast-radius queries work across substrates. Region detection is a
best-effort probe of the instance metadata service that runs once per
session and is a silent no-op on bare metal.

The process inventory collector is the "what's actually running here"
view for bare-metal and non-systemd hosts: containers with a bare init,
\*BSD, macOS, Windows, sysvinit, and openrc. It reads the process table
directly and projects each long-running process onto a service node in
your graph. Questions like "is postgres up on this box?" work even
where the Services collector has nothing to report.

To keep the graph readable, the collector only inventories daemon-like
processes. It admits a process when the init process is its parent, or
when it has been running for at least ten minutes. Transient shells,
cron one-shots, and build subprocesses never make it in.

Processes that share a name collapse onto a single node — for example,
an nginx master and its workers, or a postgres backend pool. The
longest-running instance represents the group, so attributes don't
churn as short-lived siblings come and go.

On a systemd host, the unit and the process are tracked as separate
nodes. The Services collector owns the unit's lifecycle (active state,
restarts), and the process inventory owns the live runtime attributes
(pid, cmdline, user, start time). They complement each other; neither
overwrites the other.

## Proxy egress screen

When your assistant uses the Toolkit's HTTP proxy to reach a
cloud-supplied URL, the request is dialed from **inside** your network.
The egress screen stops that path from being turned into an SSRF pivot
at cloud instance-metadata or other sensitive addresses. The screen
runs post-DNS against the resolved IP, so a hostname that resolves into
a denied range is caught too.

The screen has three layers:

* **Unconditional floor** — `169.254.0.0/16` (IPv4 link-local, including
  the AWS, GCP, and Azure IMDS address `169.254.169.254`), `fe80::/10`
  (IPv6 link-local), and `fd00:ec2::/32` (AWS IPv6 IMDS) are **always
  denied**. This cannot be disabled.
* **`proxy.block-cidrs`** — additional denied ranges layered on top of
  the floor. Use this to block an internal admin subnet or loopback that
  the proxy should never reach.
* **`proxy.allow-cidrs`** — when non-empty, switches the screen into a
  strict allow-list: a target IP must fall inside one of these ranges or
  the dial is refused. RFC1918 and loopback are **not** denied by
  default — the proxy is meant to reach internal services — so strict
  egress is opt-in here.

Deny always wins over allow. An IP in any blocked range is refused even
if it also matches `proxy.allow-cidrs`. A malformed CIDR fails the
Toolkit at startup rather than silently degrading the screen.

### Example

```yaml theme={null}
proxy:
  # Lock the proxy down to one internal subnet, but never the admin VLAN:
  allow-cidrs:
    - 10.20.0.0/16
  block-cidrs:
    - 10.20.9.0/24        # admin VLAN — denied even though it's inside the allow range
```

### Environment variables

The same knobs are available as comma-separated environment variables
for back-compat with earlier releases:

```bash theme={null}
AO_PROXY_ALLOW_CIDRS="10.20.0.0/16"
AO_PROXY_BLOCK_CIDRS="10.20.9.0/24"
```

Leaving both empty (the default) keeps the metadata floor on with
RFC1918 and loopback reachable.

## Service metadata

When the Toolkit runs as the [cluster controller](/toolkit/install#cluster-controller),
it enriches each `Service` it emits with metadata read from the
workload's labels and annotations. Labels are preferred over
annotations, and `ao.`-prefixed keys are checked first so an explicit
AO key always wins over an incidental application label.

| Attribute           | Default keys (in order)                                                 | Normalized to                 |
| ------------------- | ----------------------------------------------------------------------- | ----------------------------- |
| `env`               | `ao.environment`, `app.kubernetes.io/environment`, `environment`, `env` | `prod`, `staging`, `dev`      |
| `tier`              | `ao.tier`, `tier`                                                       | `critical`, `standard`, `dev` |
| `owned_by` → `Team` | `ao.team`, `team`, `owner`                                              | verbatim                      |

Values are matched case-insensitively against common synonyms — for
example `production`, `prd`, and `live` all normalize to `prod`. If
no env label is set, the controller falls back to scanning the
namespace name (so `checkout-prod` still resolves to `prod`).
Unrecognized values are omitted rather than guessed.

`image` (the first container's image) and `replicas_desired` (the
spec replica count, omitted for DaemonSets) come straight from the
workload spec and need no configuration.

Override the probed keys when your fleet uses different conventions:

```yaml theme={null}
k8s:
  env-label-keys:
    - mycorp.io/environment
    - environment
  tier-label-keys:
    - mycorp.io/tier
  team-label-keys:
    - mycorp.io/team
```

Keys are tried in order; the first non-empty value wins. Leave a list
empty to keep the built-in defaults.

## Capabilities

Capabilities control whether the Toolkit on a given host will run a
**sensitive** verb at all — actions like running a shell command,
patching a Kubernetes manifest, draining a node, or writing through the
HTTP proxy. Read-only verbs (host info, package lists, metric queries,
read-only proxy requests) are never gated by this control.

Two independent gates decide whether a sensitive verb runs:

* **AO Cloud IAM** decides **who** may invoke a verb against which
  Toolkit. Manage this in AO Cloud under **IAM** (policies, roles,
  action sets, resource sets).
* **On-box `capabilities.allow` / `capabilities.deny`** decides **what
  this host will run** for anyone. Set in the Toolkit's config file
  (or via environment variables) on the host itself.

These two gates are independent. A verb runs only if IAM allows the
caller **and** the on-box policy allows the host. The on-box gate is
the un-forgeable floor: it lives on the host and a compromised control
plane cannot turn it off remotely.

### Default posture

With no on-box config, the Toolkit is **permissive on the box and
relies on IAM** to gate who can invoke a verb. Tighten an individual
host with `capabilities.deny`, or pin it to a fixed set with
`capabilities.allow`.

<Note>
  This includes the **Kubernetes cluster controller**: its shipped
  StatefulSet manifest is permissive at the capability layer too, so a
  fresh install relies on IAM to gate destructive verbs. Because the
  controller can act on the whole cluster, it's a good candidate for
  on-box hardening — see [Harden the cluster
  controller](#harden-the-cluster-controller).
</Note>

### How the on-box gate evaluates

For each sensitive verb:

```
allowed-on-this-host =
  not in deny
  AND (allow is empty  OR  matches allow)
```

* **`deny`** — always wins. A verb listed here is off, regardless of
  IAM or `allow`.
* **`allow` empty (default)** — permissive: the host will run the verb
  if IAM authorizes the caller.
* **`allow` non-empty** — **authoritative whitelist**: the host runs
  exactly the verbs matching `allow` (minus anything in `deny`) and
  nothing else. This is robust against new sensitive verbs added in
  future Toolkit releases — they stay off until you list them.

### Configure on-box rules

Entries are globs over the full `<plugin>.<verb>` name: exact
(`shell.exec`), a whole family (`proxy.*`), or everything (`*`).

Pin this host to a fixed set:

```yaml theme={null}
capabilities:
  allow:
    - "proxy.read"     # this host runs only proxy.read, nothing else
```

Or stay permissive and just turn one verb off:

```yaml theme={null}
capabilities:
  deny:
    - shell.exec       # refuse arbitrary shell exec on this host
```

Equivalent environment variables (comma-separated):

```bash theme={null}
AO_CAPABILITIES_ALLOW="proxy.*"
AO_CAPABILITIES_DENY="shell.exec"
```

A malformed glob in the on-box config fails the Toolkit at boot so you
catch the typo immediately.

### Harden the cluster controller

The Kubernetes [cluster controller](/toolkit/install#cluster-controller)
ships permissive: with no on-box capability config it relies on IAM,
same as any other host. Because one controller pod can act across the
whole cluster, you may want to add an on-box floor so destructive verbs
can't run there even if IAM would allow them. The reference manifest
ships the env hook commented out for exactly this — set
`AO_CAPABILITIES_DENY` (or `AO_CAPABILITIES_ALLOW`) on the
`k8s-controller` container.

A good baseline is to drop arbitrary shell exec — the controller never
needs an interactive shell — while leaving the mutating Kubernetes
verbs IAM-gated:

```yaml theme={null}
- name: AO_CAPABILITIES_DENY
  value: shell.exec
```

To lock it down further, switch to an authoritative allowlist so only
the verbs you name can run (everything else stays off, including
sensitive verbs added in future Toolkit releases):

```yaml theme={null}
- name: AO_CAPABILITIES_ALLOW
  value: "node.*,rollout.*,scale.set"
```

`deny` always wins over IAM and over `allow`, and these gates use the
same glob semantics described above. The Toolkit binary's own default
is unchanged and stays permissive, so Linux hosts and the per-node
DaemonSet are unaffected.

## Kubernetes observation scope

The cluster controller watches every namespace by default. Scope it to
a subset with `k8s.namespaces-allow` (allowlist) or
`k8s.namespaces-deny` (blocklist). Both empty watches everything; the
two lists are mutually exclusive.

```yaml theme={null}
k8s:
  namespaces-allow:
    - prod
    - staging
```

```yaml theme={null}
k8s:
  namespaces-deny:
    - kube-system
    - kube-public
```

Scope is enforced at the apiserver for watches **and** on every verb
the controller can run:

* **Observation** — the controller does not list or watch resources
  outside scope, so they never appear in the cluster graph.
* **Actions** — `scale.set`, `rollout.restart`, `rollout.undo`, and
  `rollout.status` against a workload in an out-of-scope namespace are
  refused with a clear "outside this controller's configured
  observation scope" error.
* **Events** — `events.list` skips out-of-scope namespaces. A pinned
  `namespace` argument outside scope is refused; a cluster-wide list
  drops out-of-scope events client-side. Cluster-scoped events (for
  example, Node events with no involved-object namespace) always pass
  through.

This is a hard boundary, not a default. If you scope a controller out
of `kube-system`, nothing routed through it — read or write — can
touch `kube-system`. Run a second controller with a different scope if
you need split coverage with different IAM around each.