> ## Documentation Index > Fetch the complete documentation index at: https://docs.automatedoperations.com/llms.txt > Use this file to discover all available pages before exploring further. # Install the Toolkit > Install the AO Toolkit on Linux hosts or Kubernetes — per-node DaemonSet for host visibility, cluster controller for apiserver topology. The Toolkit is a single static binary. It collects service state (systemd), system metrics, and infrastructure topology, and streams them to AO Cloud over an outbound mTLS channel. ## Requirements * Linux (x86\_64 or arm64) * Outbound HTTPS/gRPC to AO Cloud * Root (or equivalent) to observe system services ## Install Use the install script for the common case. The OCI options below are for air-gapped, immutable, or container-based hosts where pulling a shell script isn't an option. ### Install script ```bash theme={null} # Copy the exact command — including your enrollment token — from the dashboard curl -fsSL https://get.automatedoperations.com | sh -s -- --token ``` ### Container image The Toolkit publishes a multi-arch container image (linux/amd64 and linux/arm64) on the GitHub Container Registry. The package is public — no `docker login` required. ```bash theme={null} docker pull ghcr.io/automatedoperations/ao-toolkit: ``` Use a pinned tag (for example `v1.4.2`) in production; `:latest` tracks the newest release. Optionally verify the image's keyless cosign signature before running it on production hosts: ```bash theme={null} cosign verify ghcr.io/automatedoperations/ao-toolkit: \ --certificate-identity-regexp 'https://github.com/automatedoperations/ao-toolkit/.+' \ --certificate-oidc-issuer https://token.actions.githubusercontent.com ``` A successful verification prints `Verified OK`. ### Standalone binary (OCI artifact) Each release also publishes the raw Linux binary as an OCI artifact you can pull with [`oras`](https://oras.land) — useful for hosts that don't run a container runtime. ```bash theme={null} # linux/amd64 — replace amd64 with arm64 for arm hosts oras pull ghcr.io/automatedoperations/ao-toolkit-cli:-linux-amd64 chmod +x ao-toolkit-linux-amd64 ./ao-toolkit-linux-amd64 enroll --token ``` The Linux OCI binaries are cosign-signed against the `ao-toolkit-cli` reference. macOS and Windows builds are distributed through the dashboard with Apple notarization and Authenticode signatures. ## What enrollment does 1. The Toolkit generates a keypair **on the host** — private keys never leave the machine. 2. It exchanges your one-time enrollment token for a short-lived mTLS client certificate. 3. Certificates renew automatically in the background for the life of the install. ## Verify Your host appears in the dashboard within seconds of enrollment, reporting service status and system metrics. ## Kubernetes On Kubernetes, the Toolkit ships in two complementary shapes. Pick one or apply both — the dashboard's **Install** page renders ready-to-apply manifests for each. ### Per-node DaemonSet One Toolkit pod per node, observing the node's host services and system metrics — the same surface as the Linux install above. Each pod persists its mTLS identity to `/var/lib/ao-toolkit` on the node, so restarts re-enroll under the same key. Use the DaemonSet when you want host-level visibility (systemd units, node metrics, host topology) on the nodes themselves. It needs host write access for the identity volume, so it does **not** run on locked-down environments like GKE Autopilot or EKS Fargate. ### Cluster controller A 2-replica `StatefulSet` that watches the apiserver and emits the cluster graph — `Cluster`, `Service`, `Replica`, `ScheduledTask`, and `BatchJob` nodes, plus the Kubernetes event stream. Read-only by default through a scoped `ClusterRole`; not privileged and no host access. Each pod keeps its mTLS identity on a per-pod PVC. At the capability layer the controller is **permissive by default**, like every other host — destructive verbs are gated by IAM, not pre-denied on the box. Because one controller pod can act across the whole cluster, the reference manifest ships a commented-out `AO_CAPABILITIES_DENY` hook so you can add an on-box floor (for example denying `shell.exec`, or switching to an allowlist). See [Harden the cluster controller](/toolkit/configuration#harden-the-cluster-controller). Use the controller when you want the cluster's logical topology (Deployments, StatefulSets, CronJobs, Events) in AO Cloud — including on managed environments where the DaemonSet can't run. The controller enriches each Service with `env`, `tier`, `image`, `replicas_desired`, and an `owned_by` edge to a `Team` node — derived from labels and annotations on the workload. See [Service metadata](/toolkit/configuration#service-metadata) for the label keys it consults and how to override them. Each `Replica` carries live health signals you can query directly: `condition:ready`, `condition:oom` (sticky — stays true while the kubelet still records the OOM kill), `condition:crashloop`, `restart_count`, and `last_terminated_reason`. The controller also derives Service→Service `depends_on` edges from `NetworkPolicy` rules: ingress peers depend on the target, egress peers are its dependees. `IPBlock` peers are skipped (no Service for a CIDR). #### ClusterRole and graceful degradation The controller's `ClusterRole` rendered from the dashboard is the canonical permission set — apply it as-is for the full cluster graph. It grants read-only `get`/`list`/`watch` on the resource kinds the controller emits (`nodes`, `pods`, `namespaces`, `events`, `deployments`, `statefulsets`, `daemonsets`, `replicasets`, `cronjobs`, `jobs`, and `networkpolicies`), plus the narrow mutating verbs the execution plugins need (for example `pods/eviction` for `node.drain`, `deployments/scale` for `scale.set`). If your cluster's RBAC policy forbids granting one of these, **the controller degrades instead of failing to start**. At startup it checks each watched resource with a `SelfSubjectAccessReview` and skips the informers its ServiceAccount can't list, logging a warning like: ``` controller lacks list permission; skipping this informer — graph will omit this kind (grant the RBAC to enable it) resource=networking.k8s.io/networkpolicies ``` The rest of the graph still emits. Granting the missing permission later and restarting the controller restores the skipped kind. `networkpolicies` is required for `Service` `depends_on` edge derivation. If you carry your own ClusterRole, add `networking.k8s.io/networkpolicies` (`get`/`list`/`watch`) — without it the cluster graph still emits, but service-to-service dependency edges derived from NetworkPolicies do not. ### Host visibility for `pkg.list` and `service.status` Some verbs only answer honestly when the Toolkit can actually observe the **node's** state — not the container's. In a container, the Toolkit advertises these verbs only when the required host paths are mounted in. Without them it silently declines to advertise, and the dashboard routes the question to another Toolkit (or reports it as unsupported on that host). If you use the DaemonSet manifests rendered from the dashboard's **Install** page, the required mounts are already wired up. Adjust this only if you ship your own manifests or run the Toolkit container by hand. | Verb | Required in-container | What it observes | | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `pkg.list` | Node's package DB bind-mounted under `$AO_HOST_ROOT` — `/var/lib/dpkg` on Debian/Ubuntu nodes, `/var/lib/rpm` (and `/usr/lib/sysimage/rpm` on newer Fedora/RHEL) on rpm-based nodes | Installed packages on the node | | `service.status` | Host's DBus system bus socket bind-mounted at `/run/dbus/system_bus_socket`, plus `/run/systemd/system` visible | systemd unit state on the node | | `host.open_files` | Node's `/proc` bind-mounted under `$AO_HOST_ROOT` (Linux only) | Open file handles whose on-disk inode has drifted from the path — replaced configs still pinned in memory, held-open deleted files (rotated logs, replaced binaries) | `AO_HOST_ROOT` points the Toolkit at where the node's root filesystem is mounted inside the container (for example `/host`). It is also what `host.info` uses to read `/etc/os-release` from the node rather than from the Toolkit image. Running the Toolkit directly on the host (not in a container) needs none of this — the canonical paths already are the host's. ### Running both The DaemonSet and the controller observe different things: nodes vs. the cluster's apiserver state. Apply both to see hosts **and** the cluster graph stitched together — AO Cloud dedupes overlapping graph nodes at ingest, so there's no double-counting. The controller runs two replicas without leader election. Both replicas emit; ingest collapses duplicates by graph identity. ## Reporting a crash Public Toolkit binaries are shipped with obfuscated symbols, so a panic stack trace will show mangled function names like `main.gHpVYtB`. That's expected. When you open a support ticket, include: * The **full** panic output (the entire `goroutine` dump, not just the first line). * The release tag the host is running (`ao-toolkit version`). * The host's OS and architecture (for example `linux/amd64`). AO support uses the per-build seed shipped with each release to decode the trace back to real function names and line numbers.