OpenAI’s Jalapeño chip is built for fast inference at scale, benchmarks show

At the Hot Chips league connected Tuesday, OpenAI shared a much elaborate look astatine Jalapeño, including the archetypal batch of benchmark results for the caller system. Tested connected Semianalysis’s InferenceX benchmark, Jalapeño registered some much tokens per idiosyncratic and much throughput per kilowatt than the presently disposable state-of-the-art inference processors.

“The bottommost enactment is that the results amusement a very, precise important show beforehand implicit authorities of the art,” said Richard Ho, OpenAI’s caput of hardware, successful a property call. “Jalapeño tin service much AI enactment per portion of power, portion besides returning responses much quickly. It’s precise businesslike to service a batch of customers, but it tin besides beryllium precise debased latency.”

Notably, that examination is against an Nvidia Blackwell strategy — but by the clip Jalapeño reaches afloat deployment, the contention whitethorn person precocious significantly. Ho estimated that Jalapeño would deploy astatine the extremity of 2026 “in precise tiny volumes,” with much important deployment coming successful 2027.

First announced past October, Jalapeño was developed by OpenAI successful adjacent collaboration with Broadcom, with OpenAI’s ain models assisting successful the improvement process. The institution plans to marque Jalapeño a multigenerational platform, allowing AI products, models, chips and representation each developed successful concert.

Because of that full-stack approach, OpenAI was capable to code circumstantial phases successful the inference process that often origin friction during inference processing. In particular, Jalapeño is designed to minimize delays during the prefill and connection phases of processing, which OpenAI says often enactment arsenic bottlenecks.

“We designed Jalapeño to minimize information question and connection delays,” the institution said successful a blog station presenting the results. “This means that exemplary state, including the KV cache utilized portion generating a response, tin beryllium explicitly placed and kept section portion the strategy activates the close operation of compute, memory, and networking for each inference phase.”