Deploying an AI Agent: How I Shipped Credential Sentinel to Production

The agent was finished and it reasoned well, and that is the exact point where I get curious about the other half. Deploying an AI agent to production is a different kind of problem than building one, with its own puzzles, and I enjoy that half as much as the modeling. So once Credential Sentinel worked, I kept going and turned it into something that runs the way real software runs: containerized, orchestrated, observable, and living behind a real public address.

The first step was packaging. I wrote a Dockerfile for each service so the app carries its own Python, its own dependencies, and its own config, and behaves the same on any machine instead of only on mine. The frontend build is a multi-stage one, which is the small trick that keeps the image honest. Node, the compiler, and the dev dependencies all live in an early stage that gets thrown away, and the final image copies only the compiled Next.js output. With standalone mode turned on, that runtime image lands roughly ten times smaller than a naive single-stage build. Smaller images push and pull faster when you deploy, and they carry less that can be exploited, which for a security project felt like the right kind of detail to sweat.

Locally, Docker Compose brings the whole thing up with one command: backend, frontend, and an optional model server that only starts when I ask for it, because it wants a GPU most laptops do not have. That model server is my favorite piece of the deployment. The agent talks to its LLM through an OpenAI-compatible client aimed at a single environment variable. Point that variable at the hosted Nebius API and it uses a Llama 3.3 70B in the cloud; point it at a self-hosted vLLM instance on my own GPU and it uses that instead. Same client, same JSON, one line of config, zero code changes. vLLM earns its place because of what it does under load: PagedAttention manages GPU memory the way an operating system manages RAM, and continuous batching folds new requests into a batch mid-generation, so P95 latency stays flat as concurrency climbs instead of queueing up behind a naive server.

For production I wrote the Kubernetes manifests, and the interesting parts were all the places where reality refused to be tidy. SQLite has exactly one writer, so the backend is pinned to a single replica with a Recreate rollout, because two pods writing the same file is how you corrupt a database at a bad hour. The vLLM pod has to land on a GPU node, and GPU nodes are usually tainted so ordinary pods do not squat on expensive hardware, so the manifest carries both a node selector and a matching toleration to be allowed there. And the ingress needed special handling, because the agent streams its reasoning over Server-Sent Events for minutes at a time, and most proxies kill a connection after sixty seconds. A few timeout annotations and a buffering switch keep that long stream alive instead of silently dropping the UI’s connection mid-run.

None of that is trustworthy if you cannot see it, so the backend exposes a Prometheus metrics endpoint. Three lines of instrumentation give me request counts, an in-flight gauge, and a latency histogram I can slice into P50, P95, and P99. Logs tell me what happened after something breaks; metrics let me set an alert that pages me when P95 drifts past five seconds, before anyone has to notice.

And then the satisfying part: it is live. Fly.io takes the same Dockerfile and turns it into a real server at a public URL in about five commands, with no VPC or IAM or load balancer to wrestle first, and the frontend sits on Vercel pointed at the backend. It is not the answer I would reach for under heavy production traffic. There, AWS or GKE with a managed Postgres would be the honest choice, and I know exactly what that migration looks like, since LangGraph’s Postgres checkpointer is a near drop-in that would let the backend scale past its single replica. But for getting the whole system running end to end, where I can actually use it and keep tinkering, Fly.io is strictly the right call.

The build was where the idea came to life. The deployment is where it became a system I could actually run, and I found the second half every bit as satisfying as the first.

If you want the design story behind the agent, it is in the build writeup, and the part where I tried to break it on purpose is in the evaluation post. Code is here if you want to poke at it: github.com/rashmi1112/Credential-Sentinel.