Every ML team in your organization is writing their own inference setup script. Here's the alternative.
You have a script that works. It installs vLLM, downloads the model from object storage, sets the port, and starts the server. You wrote it six months ago for Team Alpha's first inference environment. It took a day. It worked for the first time. You documented it in Confluence and moved on.
Last week, Team Delta tried to use it on Azure. It broke because the object storage path format is different and the startup timing assumptions don't hold on Azure's slower cold-start instances. Team Beta's version works but uses a different inference server, and their port configuration conflicts with the security group rules you standardized last month. Team Gamma deployed something three weeks ago that you found out about when the security team did a shadow infrastructure audit.
The script still works. The problem is that you now have five versions of it, none of them are yours, and you have no operational visibility into any of them.
The instinct is to write better documentation. A more detailed runbook. A README that covers the Azure edge cases. A Confluence page with the right variable names per provider.
The documentation will get ignored — not because your team is careless, but because the person setting up an inference environment at 3pm on a sprint deadline will take the path of least resistance. If the path of least resistance is "clone the script, change the model URL, run it," that's what they'll do. Governance that depends on people reading documentation is just aspiration.
The root cause isn't documentation. It's that inference deployment doesn't have a consistently enforced platform primitive. Compute has one — your team provisions VMs through a standard interface. Networking has one — there are approved patterns and tools. Inference environments, in most organizations, fall back to: a folder of shell scripts, a Confluence page, and a verbal agreement that teams will ask before they deploy.
You can only govern what flows through controlled interfaces. The moment a team copies a script into their own repo and runs it independently, that deployment is outside your control and scope of governance.
The consequences are predictable: inconsistent runtime configurations, orphaned instances nobody knows are running, port and dependency conflicts across teams, no audit trail, no access controls, no cost attribution. All of it shows up as your problem because you're the platform team.
Here's a representative version of the script that's probably somewhere in a repo near you:
#!/bin/bash
# TODO: make this work on Azure (ask platform team?)
# Updated by @jsmith to use Llama 3 — don't change model URL
pip install vllm==0.3.2 # pinned because 0.4.x breaks on our CUDA version
aws s3 cp s3://prod-models/llama3-8b/ ./model/ --recursive
# hardcoded — don't change unless you know what you're doing
export MODEL_PATH="./model"
export PORT="8000" # Beta team uses 8001, Gamma uses 8080
export GPU_MEMORY_UTILIZATION="0.95" # OOMs on g4dn, use 0.85 on g4dn
python -m vllm.entrypoints.openai.api_server \
--model "$MODEL_PATH" \
--port "$PORT" \
--gpu-memory-utilization "$GPU_MEMORY_UTILIZATION"
# Delta team: if this fails on Azure, ask @jsmith or check #platform-help
This script works. It also carries four months of operational debt in its comments: a pinned library version because something upstream broke, hardcoded credentials path, cross-team port conflicts documented as inline comments, GPU memory values that differ by instance type with no enforcement mechanism.
Multiply this by the number of ML teams in your organization. Then multiply by the number of models each team is serving. Then consider what happens when someone changes the model URL, the inference server version, or the GPU type — and the script that was working yesterday silently breaks overnight.
"Governance" in a platform engineering context means something specific: a system in which the deployment decision and the operational constraints are separated. The person deploying doesn't need to know the right memory utilization for a g5.xlarge — they need a deployment option that has the right value baked in. The platform team sets the constraints. The application team fills in the parameters that are genuinely theirs to decide.
That separation is how you scale. It's how Kubernetes works — RBAC and resource quotas aren't there to slow teams down, they're there so you don't need to review every deployment manually. Inference environments need the same model: a controlled set of approved configurations that teams can self-serve, with guardrails that prevent the classes of mistakes that end up as 2am incidents.
emma's inference workflows give platform teams a concrete implementation of this model.
Step 01 — You author the template (platform team owns this)
Define the workflow in emma: VM size limits (minimum and maximum instance types the template will allow), the shell script that runs on startup (install runtimes, pull the model, configure the server, start the endpoint), and the parameterized inputs — the things that are genuinely variable per deployment, like model URL, port number, environment flags.You can fix everything else at the platform level.
Step 02 — Set the access controls (RBAC enforced at deploy time)
Configure RBAC — which teams or roles can deploy this template, and which can only view it. Manage Infrastructure rights are required to deploy; you control who has them. Different templates can have different access scopes: a GPU inference template for production workloads has tighter access than a CPU template for experimentation.
Step 03 — Publish to the workflow library (Draft → Published lifecycle)
Move the template from Draft to Published. It appears in the workflow library for authorized teams.
Step 04 — Teams self-serve — within your guardrails (app teams self-serve)
An ML engineer browses the library, selects the template, fills in the parameters (their model URL, their port), and deploys. The VM provisions, the script runs, the endpoint comes up. They never touched the instance configuration, the runtime version, or the GPU memory settings. The deployed instance appears in emma as a standard VM — visible, manageable, cost-attributed.
Here’s how it happens in practice:
Step 05 — You have visibility over all of it (audit trail built in)
Every deployment through a workflow template is logged: who deployed it, which template version, which provider, which region, when. Deployment status, execution logs, and cost preview are built into the workflow view. When the security team asks "what inference environments are running and who deployed them?" — you can answer that in minutes.
# Template: GPU Inference — Single Model
# Platform team owns this. Teams fill in the parameters below.
instance_constraints:
allowed_types: [g5.xlarge, g5.2xlarge, g5.4xlarge]
default: g5.xlarge
max_concurrent_per_team: 3
parameters: # teams fill these in at deploy time
model_url: # s3:// or https:// path to model artifacts
port: # default 8000, validated against reserved list
env_flags: # optional: extra vLLM flags
startup_script: # platform team manages this
pip install vllm==0.4.1 --quiet
model_fetch "{{ model_url }}" ./model/
export GPU_MEMORY_UTILIZATION=0.90 # validated for g5 family
python -m vllm.entrypoints.openai.api_server \
--model ./model --port "{{ port }}" "{{ env_flags }}"
rbac:
deploy: [ml-engineers, platform-team]
view: [all-engineers]
version: 3
status: Published
Compare that to the script from the top of this post. Same outcome — an inference server running with the right model. Different operational model: the constraints that cause incidents (instance type, GPU memory, runtime version, port validation) are no longer in comments. They're enforced.
Writing and maintaining scripts positions your team as a service provider — teams come to you for the script, fork it, and the moment they do, your ability to enforce changes depends entirely on them pulling the latest version. Publishing governed workflow templates positions you as a platform team — other teams deploy within constraints you define, and operational visibility is built in from the start.
The practical difference is in how updates propagate. When you change the startup script in a workflow template — to upgrade the inference server version, or tune GPU memory for a new instance type — that change is in one place. Every team that deploys after that point gets the new behaviour automatically. You're not hunting forks or opening PRs and hoping people pull before their next deployment.
Even if that doesn't solve running instances until someone redeploys, it does mean the problem shrinks over time rather than compounding.
The security and compliance difference is equally concrete. Right now, if a compliance team asks for an inventory of all running inference environments, you're doing archaeology — digging through personal repos, undocumented forks, and configurations that may belong to people who've left the company.
With emma’s workflow-based deployments, every instance is traceable to the template it was deployed from, the user who deployed it, and a timestamp. Cost is attributed per VM, project, and provider from the moment of deployment. It’s not only useful for audits, but also becomes the operational baseline for running AI infrastructure at scale.
Platform engineering's job is to give application teams a safe, fast path to the infrastructure they need — and to make the unsafe path harder than the safe one. Inference environments are a category where the path of least resistance is still "here's a script, good luck."
The teams deploying inference environments aren't going to stop. AI is too central to what product teams are building for the rate to go down. The question is whether those deployments happen inside your governance model or outside it. Right now, for most organizations, they happen outside it — in personal repos, undocumented forks, and configurations that belong to people who may have left the company.
Workflow templates aren't a silver bullet. They're primitive — the same kind of primitive that made Kubernetes governance tractable. The script chaos is a symptom. The absence of a governed deployment primitive is the cause. The organizations that govern this now won't be the ones explaining to finance why GPU spend doubled with no attribution or answering compliance team's inventory request with a spreadsheet built from memory.
Inference workflows are one part of emma's AI infrastructure layer — which also covers GPU compute across five providers, managed Kubernetes, cross-cloud networking, observability, and unified cost visibility under one control plane. Learn more in the emma AI capabilities brief →
See governed inference in a live demo — template authoring, RBAC configuration, self-service deployment, and the audit trail, end to end in 30 minutes. Book a demo at emma.ms
.avif)
