Guides
Build or buy your AI control layer
David Buxton · · 3 min read
Somewhere in your company there is an engineer who could set up a model gateway by Friday. Keys in one place, requests routed to whichever provider is cheapest or best, a dashboard of spend. Some of the best-run mid-market companies we talk to have exactly that. It is a good piece of work, and about a tenth of the problem.
Here is where the build-versus-buy line sits for a company with a capable IT team and no platform team.
What the gateway solves
A gateway decides where each request goes. Route by model, by cost, by fallback. Hold the API keys in one place instead of on people’s personal cards. See the token count by key.
Open-source options do this well, and if your engineers run one, keep it. Nothing below replaces it. The mistake is treating the routing layer as the platform when it is the plumbing under the platform.
What is left when the gateway is done
Getting it onto every laptop
A gateway serves code. Your employees need an assistant on their device that signs in with their work identity, arrives through the same device management as everything else, and works on day one without a ticket. That means a desktop application, packaging for three operating systems, and integration with your identity provider. It is the part that turns an API into adoption, and it is a rollout of its own.
The first workflows
The technical teams will build their own. Everyone else needs three to five real workflows built with them, end to end, before they see the point. That is people work more than engineering work, and it is where adoption is won or lost. A gateway has nothing to say about it.
Skills that travel, with someone reviewing them
Once people build workflows, the good ones need to spread and the risky ones need a gate. A skill that reads the CRM, drafts an email and sends it has to be reviewed by someone who owns that process before it runs for the whole team. That means a library, versions, a review queue, and scanning of skills and connectors before they run. Building that well is a product in itself, which is why skill sharing is a feature and not a folder.
Policy people can rely on
Which team gets which models. What each team can spend before it drops to a cheaper model rather than losing access. Which region regulated data is processed in. These have to be enforced, not requested, and they have to be visible to the person who owns the policy without asking an engineer.
The audit trail
The security review, the ISO 42001 assessment and the customer questionnaire all ask what the AI did, with whose data, through which tool. Token counts at the gateway do not answer that. You need the prompt, the model, the tool calls and the outcome, exportable, which is what an auditor expects to see.
Keeping up
Every provider changes its API, its models and its pricing several times a year. Operating systems and identity providers change too. An internal platform is never finished. Whoever built it maintains it, and when they leave, the knowledge leaves with them.
Where the line sits
If you have a platform team whose job is to build internal tools for staff, and the AI platform is on their roadmap, build it. Some companies do, and they build good ones.
If you have a strong IT team with other work to do, buy the platform and keep your gateway. The gateway is the one commodity part. The rest is a year of product work that nobody in the company is measured on, and it will lose to whatever is measured.
Harriet is built for the second company. Bring your own keys, keep your routing if you like it, and get the desktop, the skills, the policy and the audit trail without hiring for them. Book a call and we will map what you have built against what is left.
Common questions
Should we build our own AI gateway?
A gateway that routes requests between model providers is a solved problem, and a good engineer can run one in days. The question is everything around it. Getting an assistant onto every device, building the first workflows with non-technical teams, reviewing and sharing skills, logging every tool call, and enforcing regions and budgets. That is a product, and unless you have a team whose job is to build products for your own staff, it is what you should buy.
What is the difference between an LLM gateway and an AI control plane?
A gateway sits between your code and the model providers, handling keys, routing and rate limits. A control plane covers the whole path from an employee's laptop to the model and back. Provisioning, identity, per-team model policy, spend caps, skill and connector review, and an audit trail of what each request did. A gateway is one component of a control plane.
We already run LiteLLM or a similar gateway. Do we still need anything?
The gateway covers routing and keys, and it can stay. What it does not cover is the employee side. Who gets what model, how the assistant reaches every device, how a workflow built by one person reaches the team, and what the security review needs to see. Companies with a gateway in place tend to buy for the rollout, not the routing.
How long does it take to build an internal AI platform?
The gateway, a week or two. A desktop assistant that installs through your identity provider, a skill library with review, per-team policy and budgets, and audit logging, closer to a year of one or two engineers, plus ongoing maintenance as every provider changes its API. Most mid-sized companies do not have those engineers to spare, and the ones that do have other things for them to build.