Who ax is for#
AI researchers running evaluation and training loops
AX lets you spin up hundreds of reproducible sandboxes with exact toolchain and dataset configurations to collect agent trajectories, run reinforcement learning loops, and evaluate models at scale. The suspend/resume mechanic means long-running experiments persist across interruptions without losing state.
Skip if:
Your experiments fit on a single machine or a simple Kubernetes Job with no suspend/resume requirement. AX adds infrastructure overhead that isn't justified for single-node workloads.
Teams building autonomous coding agents
AX sandboxes provide CPU and memory limits, pre-wired Git workspace setup, and outbound network fencing, which matches the exact requirements of a coding agent: isolated execution of untrusted code, access to a specific repository branch, and controlled tool access.
Skip if:
You are building a chat-over-code assistant rather than a fully autonomous agent that runs builds and executes commands. The full AX stack is heavier than needed for stateless LLM interactions.
Platform engineers replacing bespoke agent lifecycle glue
Teams that have built Kubernetes-based lifecycle management for their agentic systems can replace custom suspend/resume and workspace provisioning logic with AX's native primitives. The kubectl-shaped API means existing GitOps and CI/CD tooling applies directly.
Skip if:
Your agents are short-lived, stateless, or fit naturally into standard Kubernetes Jobs. AX's value concentrates in long-lived stateful actors; stateless workloads don't benefit from the additional layer.
The problem it solves#
Running autonomous agents at scale exposes a fundamental mismatch with existing infrastructure. Kubernetes Jobs and cloud batch services are designed for stateless, run-to-completion tasks: they have no concept of an agent that runs for a few seconds, waits an hour for a model response, then resumes. Keeping an idle sandbox alive for the full wait wastes compute; tearing it down loses state.
Traditional orchestrators also lack built-in support for network isolation around untrusted agent code, pre-wired tooling workspaces, or centralized credential rotation for LLM providers. Teams building agentic systems end up writing bespoke lifecycle management on top of Kubernetes, each time re-solving the same suspend, resume, isolation, and workspace-provisioning problems. Researchers running trajectory collection or reinforcement learning evaluation loops face the same issue at higher volume: reproducing exact sandbox environments with specific toolchains requires infrastructure work that has nothing to do with the research.
How it solves it#
Declarative YAML task manifests
Define tasks, workspaces, network gateways, and model configuration as ax.io/v1alpha1 YAML manifests and apply them with a single command. The same file can include multiple document types; AX wires them together at apply time. The format follows the Kubernetes convention, so existing YAML tooling and linting pipelines apply directly.
Sub-second suspend and resume
Idle agents waiting on model responses, external tool calls, or human approval are checkpointed and suspended automatically. Resumption takes under a second with no cold-start delay. This is handled by the underlying Agent Substrate runtime and is transparent to the agent code; calling ax resume picks up execution from the exact point it was paused.
Pre-wired workspace setup
Workspaces declare the Git repositories, MCP servers, and skill packages an agent needs. AX sets up every sandbox from the workspace definition before the task starts, so the agent begins with a warm environment. You can also describe a workspace goal in plain English and AX hands it to a setup agent to install the right toolchain on first boot.
Outbound network gateway policies
Gateway manifests define an explicit allowlist of hosts and ports the task sandbox can reach. Traffic outside the allowlist is blocked. Credentials can be injected into outbound requests at the gateway level, so agent code never handles API secrets directly.
Live shell access to running sandboxes
ax ssh opens an interactive shell inside a running agent sandbox, or runs a one-off command in the sandbox. Tasks need spec.debug: true enabled to allow SSH access. It is useful for inspecting what an agent is doing mid-run, running diagnostic commands, or verifying workspace contents without interrupting execution.
Centralized LLM model configuration
A Model manifest holds the provider name, model identifier, and a reference to a Kubernetes secret containing the API key. Rotating a key or pinning a new model version is a single ax apply. All tasks in an atespace share the model configuration, so updating a fleet of agents to a new model version requires no changes to individual task definitions.
Strengths and trade-offs#
Strengths
- Kubectl-shaped CLI with familiar conventionsThe ax CLI follows the same verbs as kubectl: apply, get, describe, watch, delete. Anyone who has used Kubernetes can apply a task file and inspect its status within minutes. It also follows the active kubectx context, so switching between staging and production clusters requires no extra configuration.
- Apache-2.0 license with no usage restrictionsThe Apache-2.0 license allows use, modification, and distribution for any purpose including commercial production deployments. There are no seat limits, API call fees, or restrictions on the type of workloads you run. The entire codebase is open, so teams can fork and modify the control plane.
- Dense multiplexing reduces idle compute costDozens of tasks share worker resources, and idle time is turned into available capacity for other tasks. You pay for compute only when agents are actively executing, not while they wait for model API responses. This contrasts with most container orchestrators, which hold resources for the full task duration.
- Built from Google DeepMind agentic runtime researchAX originated from internal Google research into agentic execution engines and the specific lifecycle requirements of stateful, long-running agents. The underlying Agent Substrate runtime was designed from the ground up for massive density and fast stateful actor lifecycles, not adapted from microservice or batch infrastructure.
Trade-offs
- -Breaking changes expected before stable releaseThe README carries an explicit warning: the team is still refining core concepts, protocols, and specifications, and will likely introduce major breaking changes before a stable release. Pinning a specific commit and testing upgrades in staging is necessary for any deployment that cannot tolerate breakage.
- -Requires Kubernetes, Agent Substrate, and a container registrySetup requires an existing Kubernetes cluster, ko for image builds, a container registry the cluster can pull from, and a running Agent Substrate Control API. This is not a Docker Compose or single-binary install. Teams without existing Kubernetes infrastructure will need to stand it up before running their first task.
- -Self-hosted only, no managed optionAX is self-hosted only. There is no hosted control plane, managed cloud tier, or one-click deployment option. All infrastructure management, upgrades, and operational concerns are the deploying team's responsibility.
ax vs alternatives#
AX vs Modal
AX and Modal both target teams running AI workloads or agents at scale. The key difference is deployment model: AX is self-hosted on your Kubernetes cluster under Apache-2.0, while Modal is a managed cloud service with per-compute-second billing.
| Feature | AX | Modal |
|---|---|---|
| License | Apache-2.0 | Proprietary |
| Self-hosting | Yes (required) | No |
| Stateful suspend/resume | Yes, sub-second | No native equivalent |
| Network isolation | Gateway manifests | Sandbox isolation |
| Workspace pre-wiring | Git repos, MCP servers | Custom image builds |
| Pricing | Infrastructure cost only | Per-compute-second |
AX is the better choice when you need data residency, control over the execution environment for compliance or security reasons, or workloads that require long-lived stateful actors with sub-second suspend and resume. The Gateway and Workspace primitives are built specifically for agentic systems, not general serverless functions.
Modal is worth considering when you want to skip Kubernetes setup entirely and run Python functions or containerized agents on a managed service with simpler onboarding. Modal targets Python developers and requires no cluster operations; AX requires Kubernetes expertise and ongoing operational investment.
AX vs AWS ECS Fargate
For teams running agent workloads on AWS, Fargate manages container scheduling on managed infrastructure. Fargate works well for short-lived or stateless containers but has no native concept of an agent that suspends mid-execution and resumes later. Building suspend/resume, workspace pre-wiring, and network allowlist management on top of Fargate requires significant custom tooling.
AX is worth the infrastructure cost when your agents are long-lived, stateful, or require fast resume after idle periods. Fargate is the simpler path for stateless or short-running agent executions where the full AX control plane adds more complexity than the workload justifies.
Quick start#
Install the AX CLI via Go, then deploy the control plane to a Kubernetes cluster.
```bash
go install github.com/google/ax/cmd/ax@latest
make deploy AX_IMAGE_REPO=<your-registry>
```What it's built on#
- Languages
- GoPython
FAQ#
Does AX require Kubernetes to run?
Yes. AX runs on top of Agent Substrate, which requires a Kubernetes cluster. You also need ko for building images, a container registry, and a reachable Agent Substrate Control API endpoint. There is no single-binary or Docker Compose path. The README quickstart assumes you have an existing cluster and walks through the make deploy command.
What is Agent Substrate and do I need to install it separately?
Agent Substrate is the underlying compute runtime that handles sandboxed execution and stateful actor scheduling. AX builds on top of it and requires a reachable Agent Substrate Control API (in-cluster default: api.ate-system.svc.cluster.local:443). The AX quickstart assumes Agent Substrate is already running; see the Agent Substrate repository for setup instructions.
Is AX production-ready?
Not yet. The README explicitly warns that the team is still refining core concepts, protocols, and specifications, and that major breaking changes are likely before a stable release. It is appropriate for research and experimental production workloads where you can manage upgrades carefully, but not for systems that require API stability guarantees.
How does AX handle LLM API credentials?
A Model manifest holds a reference to a Kubernetes secret containing the API key. Agent code never handles credentials directly; AX injects them into requests at the infrastructure level. Rotating a key means updating the Kubernetes secret and running ax apply on the model manifest. All tasks in an atespace share the model configuration.
Can AX run non-agent workloads like Jupyter notebooks?
Yes. The website lists Jupyter notebooks, interactive coding agents, long-running agent servers, and headless browser testing as supported workloads. AX's sandbox is general-purpose; the Task primitive runs any containerized process with the workspace and network configuration you specify.
Similar open-source tools#
FckSignups
Open-source tools that work instantly, no signup required
agent-toolkit-for-aws
Empower AI agents to build and manage AWS applications
Omnara
Open-source agent deployment API. Self-host or use Omnara Cloud.
substrate
Kubernetes-native sandbox runtime for AI agent execution
orbi
GitHub Issues in, reviewed code merged, tagged releases out
dify
Visual AI workflow builder with RAG, agents, and self-hosting