Together AI has launched Together Compiler (TKCC), an agent-driven system designed to perform continuous optimization on inference serving engines. The framework uses AI coding agents to rewrite model forward passes dynamically in response to shifting production traffic profiles, request lengths, and hardware configurations without requiring full engine restarts.
To overcome slow iteration loops, Together Compiler runs candidate forward passes over RPC and utilizes a two-tier measurement process for fast evaluation. Together AI reports that this approach yielded a 252-fold increase in agent experiment throughput compared to engine-restart baselines. On the SemiAnalysis InferenceX AgentX benchmark, the system achieved up to 2.6 times higher throughput per user compared to top open configurations.
The continuous compilation loop addresses real-world inference challenges where static GPU kernel tuning fails due to variable input lengths and Mixture-of-Experts expert imbalances. Beyond large language models, the compiler improved throughput across eight scientific models by 1.5x to 6.4x while maintaining bit-exact outputs.
Why it matters
Dynamic continuous compilation reduces serving latency and cost by adapting GPU execution graphs to real-time production traffic drifts.
RPC-based forward pass swapping enables high-throughput agentic optimization loops without incurring expensive engine cold restarts.
Automated coding agents operating on raw engine source outpaced expert-curated optimization strategies in complex model serving setups.
Source: marktechpost.com



