What belongs in a common AI platform for 200 teams?

The boundary I would draw is around facts the platform can enforce for everyone. Identity, tenant isolation, credentials, admission, usage accounting, policy at tool boundaries, and a way to trace a request belong there. Whether a passage answers a finance question, whether a refund is justified, and whether a medical summary is clinically sound belong with the team that owns that product. The platform can make those decisions easier to test. It cannot make them once for two hundred unrelated domains.

Here is the first pass I would put in front of the teams:

This is a contract, not four giant services every team must call. A team with a simple summarizer may use the gateway and eval harness without adopting a workflow engine. A team with a custom search stack may publish the same source identity and permission contract through an adapter. The platform should make the common path cheap to use, with stable SDKs and reference implementations. It should not make teams lose a needed capability behind a lowest common denominator API.

At the gateway, I would keep provider secrets and tenant policy away from application code. Each request carries a workload identity, end user or delegated principal when needed, tenant, data classification, permitted regions, budget, and deadline. The gateway checks the applicable policy, records the selected model and effective configuration, and returns an explicit error if a model cannot satisfy it. Routing can use cost and capacity, but it cannot silently substitute a model with different tool behavior or data handling terms when the app depends on those. Give the app a capability contract and a way to pin or constrain the route for a verified release.

QPS is a weak fairness measure here. One request may contain 200 tokens and another 200,000. I would meter input, output, cached and reserved capacity as appropriate for each serving pool, set tenant budgets and admission rules, and watch queueing and tail latency by class. A global rate limit that lets one team's long contexts fill the shared scheduler will look fair in request counts while starving other teams. The platform owns that shared capacity problem. The application owns reducing its context when it is wasting the budget.

Retrieval needs an equally clear security boundary. The platform can provide storage, indexing, filtering primitives, and a current authorization check before source text reaches reranking or the model. The application knows which sources are authoritative and how to reconcile contradictory or dated documents. A universal chunker or embedding model will not solve both source code and policy manuals. The platform should allow versioned pipelines, expose provenance, and make deletion and permission changes testable. If it only returns opaque text blobs, the app cannot reliably explain an answer or a stale citation.

For agents, a tool call is a proposed effect, not just a model completion. The runtime can persist the operation, enforce the declared permission and approval boundary, and reconcile an unknown result after a crash. The application defines the business action and its exact approval payload. “Approved action” is meaningless if the platform does not know what amount, target, and effect were approved. Domain owners review those semantics, while platform security reviews the reusable enforcement path.

I would run this as a product. Start with a few teams whose workloads differ. Publish versioned contracts, migration windows, SLOs for shared components, and the incident owner for each boundary. Track adoption and escape hatches, but also ask whether teams can actually diagnose a bad answer or a delayed tool call using the trace they get. An internal platform with a high adoption number and no credible owner for a cross-layer failure is not finished.

The interviewer asks for an exception: a regulated team cannot send prompts or traces to the default region. I would offer an isolated data plane in the allowed region if the organization can operate one, using the same identity, policy, and telemetry schema with local storage and retention. Review the provider, subprocessors, logs, backups, and human access. If the platform cannot meet those requirements, the team gets an approved alternative path with equivalent control evidence. An “exception” that bypasses tenant policy and leaves no accountable operator is not an exception process. It is a hole.

Then the interviewer says that the regulated team wants raw provider credentials because the gateway is too slow. First measure the gateway contribution against model queueing and prefill. If the gateway is the bottleneck, fix or colocate it. If there is a genuinely incompatible latency target, a direct path can still use scoped workload credentials, quotas, region restrictions, and central usage and audit controls. The same controls may be enforced through a different topology. Handing out one shared unrestricted key would make the exception impossible to attribute or contain.

One last push: a provider changes its tool call format. Who fixes the 200 apps? The gateway and SDK should expose a versioned normalized contract where normalization is actually sound, and tests should cover the provider edge cases. The platform stages a new adapter and shows which apps depend on changed behavior. The application still tests its action semantics before switching. A silent provider swap might keep the JSON parseable and change what an approved action means. That is a platform migration with app signoff, not a hidden routing tweak.

OpenAI's platform RBAC documentation is one concrete example of organization and project scoped permissions. It does not prescribe this multi-team architecture. The ownership and exception boundaries above are the design choices for the question.