Why AI investigations are local-only in Tellagen

We are making AI investigations in Tellagen local-only.

The agent runs where the user already works: Claude Code, Codex, or another MCP client. Tellagen provides incident context, runbooks, datasource metadata, trace storage, findings, and the incident record. The agent uses local access to Grafana, Kubernetes, GitHub, logs, docs, and whatever else the team has wired up.

The cloud alternative is a hosted control plane with a customer-run agent inside the user's infrastructure. I see the appeal, but it asks teams to deploy infrastructure before they know whether the investigation is useful.

The cloud agent got heavy fast

A customer-run agent sounds simple: deploy it, connect it back to Tellagen, send jobs to it, stream results into the incident.

Reality shows up. Customers need egress rules, custom CA bundles, proxies, namespace limits, credential rotation, separate environments, Helm charts, Terraform modules, audit controls, and upgrade guidance. Every team has a different answer.

For a small SaaS team, that setup may never get tested. If the first step is installing an agent and debugging egress, the feature has already lost. The first version should fit how teams already investigate: open the local AI client, load the incident, and use the credentials already on the machine.

Local mode fits the first version

Local MCP mode is boring in the useful way. If an engineer can use kubectl, Grafana, and GitHub locally, the AI client can usually use the same tools. Tellagen does not need production observability credentials just to organize an investigation.

Tellagen should own the shared record: context, scope, runbooks, datasource names, trace steps, findings, and timeline updates. The local agent should own the investigation work. That line is safer by default.

Metadata and runbooks

The local agent still needs datasource names and usage rules. So datasource setup in Tellagen is metadata only: Grafana URLs, datasource names, required environment variables, Kubernetes namespaces, MCP server names, and the runbook to follow. Tellagen should not store Grafana tokens, kubeconfigs, API keys, passwords, cookies, Authorization headers, or username/password URLs for this mode.

The agent can query Tellagen for metadata, then check the user's local environment. If GRAFANA_URL or GRAFANA_SERVICE_ACCOUNT_TOKEN is missing, it should say so without printing the secret value.

Credentials stay local. Runbooks are shared.

Proxy later, maybe

Tellagen may eventually proxy Grafana or other systems for managed setups. That should be a separate custody mode.

A proxy needs encryption, audit logs, read-only endpoints, rate and query limits, tenant isolation, secret rotation, and careful Kubernetes restrictions. It also changes the trust boundary because Tellagen would hold or resolve credentials that can read production observability data.

I do not want that to be the first version we ask users to trust or operate.

For now, local-only investigations are the right first version: faster to try, lighter to support, and fewer secrets in Tellagen. Start an investigation, let the local AI client use existing tools, and save the useful evidence back to the incident.