The Toolkit is a single static binary. It collects service state
(systemd), system metrics, and infrastructure topology, and streams them
to AO Cloud over an outbound mTLS channel.
Requirements
- Linux (x86_64 or arm64)
- Outbound HTTPS/gRPC to AO Cloud
- Root (or equivalent) to observe system services
Install
Use the install script for the common case. The OCI options below are
for air-gapped, immutable, or container-based hosts where pulling a
shell script isn’t an option.
Install script
Container image
The Toolkit publishes a multi-arch container image (linux/amd64 and
linux/arm64) on the GitHub Container Registry. The package is public —
no docker login required.
Use a pinned tag (for example v1.4.2) in production; :latest tracks
the newest release.
Optionally verify the image’s keyless cosign signature before running
it on production hosts:
A successful verification prints Verified OK.
Standalone binary (OCI artifact)
Each release also publishes the raw Linux binary as an OCI artifact you
can pull with oras — useful for hosts that don’t
run a container runtime.
The Linux OCI binaries are cosign-signed against the ao-toolkit-cli
reference. macOS and Windows builds are distributed through the
dashboard with Apple notarization and Authenticode signatures.
What enrollment does
- The Toolkit generates a keypair on the host — private keys never
leave the machine.
- It exchanges your one-time enrollment token for a short-lived mTLS
client certificate.
- Certificates renew automatically in the background for the life of
the install.
Verify
Your host appears in the dashboard within seconds of enrollment,
reporting service status and system metrics.
Kubernetes
On Kubernetes, the Toolkit ships in two complementary shapes. Pick one
or apply both — the dashboard’s Install page renders ready-to-apply
manifests for each.
Per-node DaemonSet
One Toolkit pod per node, observing the node’s host services and
system metrics — the same surface as the Linux install above. Each pod
persists its mTLS identity to /var/lib/ao-toolkit on the node, so
restarts re-enroll under the same key.
Use the DaemonSet when you want host-level visibility (systemd units,
node metrics, host topology) on the nodes themselves. It needs host
write access for the identity volume, so it does not run on
locked-down environments like GKE Autopilot or EKS Fargate.
Cluster controller
A 2-replica StatefulSet that watches the apiserver and emits the
cluster graph — Cluster, Service, Replica, ScheduledTask, and
BatchJob nodes, plus the Kubernetes event stream. Read-only by
default through a scoped ClusterRole; not privileged and no host
access. Each pod keeps its mTLS identity on a per-pod PVC.
At the capability layer the controller is permissive by default,
like every other host — destructive verbs are gated by IAM, not
pre-denied on the box. Because one controller pod can act across the
whole cluster, the reference manifest ships a commented-out
AO_CAPABILITIES_DENY hook so you can add an on-box floor (for example
denying shell.exec, or switching to an allowlist). See Harden the
cluster controller.
Use the controller when you want the cluster’s logical topology
(Deployments, StatefulSets, CronJobs, Events) in AO Cloud — including
on managed environments where the DaemonSet can’t run.
The controller enriches each Service with env, tier, image,
replicas_desired, and an owned_by edge to a Team node — derived
from labels and annotations on the workload. See
Service metadata for the
label keys it consults and how to override them.
Each Replica carries live health signals you can query directly:
condition:ready, condition:oom (sticky — stays true while the
kubelet still records the OOM kill), condition:crashloop,
restart_count, and last_terminated_reason. The controller also
derives Service→Service depends_on edges from NetworkPolicy
rules: ingress peers depend on the target, egress peers are its
dependees. IPBlock peers are skipped (no Service for a CIDR).
ClusterRole and graceful degradation
The controller’s ClusterRole rendered from the dashboard is the
canonical permission set — apply it as-is for the full cluster graph.
It grants read-only get/list/watch on the resource kinds the
controller emits (nodes, pods, namespaces, events,
deployments, statefulsets, daemonsets, replicasets, cronjobs,
jobs, and networkpolicies), plus the narrow mutating verbs the
execution plugins need (for example pods/eviction for node.drain,
deployments/scale for scale.set).
If your cluster’s RBAC policy forbids granting one of these,
the controller degrades instead of failing to start. At startup it
checks each watched resource with a SelfSubjectAccessReview and
skips the informers its ServiceAccount can’t list, logging a warning
like:
The rest of the graph still emits. Granting the missing permission
later and restarting the controller restores the skipped kind.
networkpolicies is required for Service depends_on edge
derivation. If you carry your own ClusterRole, add
networking.k8s.io/networkpolicies (get/list/watch) — without
it the cluster graph still emits, but service-to-service dependency
edges derived from NetworkPolicies do not.
Host visibility for pkg.list and service.status
Some verbs only answer honestly when the Toolkit can actually observe
the node’s state — not the container’s. In a container, the
Toolkit advertises these verbs only when the required host paths are
mounted in. Without them it silently declines to advertise, and the
dashboard routes the question to another Toolkit (or reports it as
unsupported on that host).
If you use the DaemonSet manifests rendered from the dashboard’s
Install page, the required mounts are already wired up. Adjust
this only if you ship your own manifests or run the Toolkit container
by hand.
AO_HOST_ROOT points the Toolkit at where the node’s root filesystem
is mounted inside the container (for example /host). It is also
what host.info uses to read /etc/os-release from the node rather
than from the Toolkit image.
Running the Toolkit directly on the host (not in a container) needs
none of this — the canonical paths already are the host’s.
Running both
The DaemonSet and the controller observe different things: nodes vs.
the cluster’s apiserver state. Apply both to see hosts and the
cluster graph stitched together — AO Cloud dedupes overlapping graph
nodes at ingest, so there’s no double-counting.
The controller runs two replicas without leader election. Both
replicas emit; ingest collapses duplicates by graph identity.
Reporting a crash
Public Toolkit binaries are shipped with obfuscated symbols, so a panic
stack trace will show mangled function names like main.gHpVYtB. That’s
expected. When you open a support ticket, include:
- The full panic output (the entire
goroutine dump, not just the
first line).
- The release tag the host is running (
ao-toolkit version).
- The host’s OS and architecture (for example
linux/amd64).
AO support uses the per-build seed shipped with each release to decode
the trace back to real function names and line numbers.