An exclusive tour of Amazon’s Trainium lab, the chip that’s won over Anthropic, OpenAI, even Apple 

Shortly aft Amazon CEO Andy Jassy announced AWS’s groundbreaking $50 cardinal concern deal with OpenAI, Amazon invited maine connected a backstage circuit of the spot improvement laboratory astatine the bosom of the deal, astatine (mostly*) its ain expense. 

Industry experts are watching Amazon’s Trainium chip, created astatine that facility, for its implications for lower-cost AI inference and, potentially, a dent successful Nvidia’s adjacent monopoly.  

Curious, I agreed to go.  

My circuit guides for the time were the lab’s director, Kristopher King (pictured beneath right) and manager of engineering Mark Carroll (below left), arsenic good arsenic the team’s PR idiosyncratic who arranged the visit, Doron Aronson (pictured with yours genuinely aboriginal successful the story). 

AWS Chip laboratory leaders Mark Carroll and Kristopher King.Image Credits:TechCrunch/Julie Bort

AWS has been Anthropic’s large unreality level since the AI lab’s aboriginal days — a narration important capable to past Anthropic aboriginal adding Microsoft arsenic a unreality spouse arsenic well, and Amazon’s increasing concern with OpenAI.

The OpenAI woody makes AWS the exclusive supplier of the exemplary maker’s caller AI cause builder, Frontier, which could go an important portion of OpenAI’s concern if agents go arsenic large arsenic Silicon Valley thinks they will. We’ll spot if that exclusivity stands precisely arsenic announced. The Financial Times reported this week that Microsoft whitethorn judge OpenAI’s woody with Amazon violates its ain woody with OpenAI, namely with Redmond getting entree to each of OpenAI’s models and tech.

What makes AWS truthful appealing to OpenAI? As portion of this deal, the unreality elephantine has agreed to proviso OpenAI with 2 gigawatts of Trainium computing capacity. This is simply a elephantine commitment, fixed that Anthropic and Amazon’s ain Bedrock work are already consuming Trainium chips faster than Amazon tin nutrient them. 

There are 1.4 cardinal Trainium chips deployed crossed each 3 generations, and Anthropic’s Claude runs connected implicit 1 cardinal of the Trainium2 chips deployed, the institution said.

It’s worthy noting that portion Trainium was primitively geared toward faster, cheaper exemplary grooming (a bigger precedence a mates of years ago), it’s present tuned and utilized for inference arsenic well. Inference — the process of really moving an AI exemplary to make responses — is presently the biggest show bottleneck successful the industry. 

Case successful point: Trainium2 handles the bulk of the inference postulation connected Amazon’s Bedrock service, which supports the gathering of AI applications by Amazon’s galore endeavor customers and allows the apps to usage aggregate models.

“Our lawsuit basal is conscionable expanding arsenic accelerated arsenic we tin get capableness retired there,” King said. “Bedrock could beryllium arsenic large arsenic EC2 1 day,” helium added, referring to AWS’s behemoth compute unreality service. 

Amazon’s Trainium3 chip.Image Credits:Amazon

Trainium vs. Nvidia

Beyond offering an alternate to Nvidia’s backlogged, hard-to-acquire GPUs, Amazon says its caller chips moving connected its caller specialty Trn3 UltraServers outgo up to 50% little to tally for comparable show than utilizing classical unreality servers. 

Along with Trainium3, released successful December, this AWS squad besides built caller Neuron switches, and Carroll says that combo is transformative.

“What that gives america is thing huge,” Carroll said. The switches let each Trainium3 spot to speech to each different spot successful a mesh configuration, reducing latency. “That’s wherefore Trainium3 is breaking each kinds of records,” peculiarly successful “price per power,” helium said. 

When trillions of tokens a time are involved, specified improvements adhd up.  

In fact, Amazon’s spot squad was lauded by Apple successful 2024. In a uncommon infinitesimal of openness for the secretive company, Apple’s manager of AI publically described however it utilized different of the team’s chips — Graviton, a low-power, ARM-based server CPU and the archetypal breakout spot this squad designed. Apple besides lauded Inferentia — a spot specifically designed for inference — and gave a motion to Trainium, which was caller astatine the time. 

These chips correspond the classical Amazon playbook: See what radical privation to buy, past physique an in-house alternate that competes connected price. 

The drawback for chips, historically, has been switching costs. Applications written for Nvidia’s chips indispensable beryllium re-architected to enactment with others — a time-consuming process that discourages developers from switching.

But the AWS spot squad proudly told maine that Trainium present supports PyTorch, a fashionable unfastened root model for gathering AI models. That includes galore of the ones hosted connected Hugging Face, a immense room wherever developers stock unfastened root models.

The transition, Carroll told me, requires “basically a one-line change, and past recompile, and past tally connected Trainium.” In different words, Amazon is attempting to spot distant astatine Nvidia’s marketplace dominance wherever possible.

AWS has besides this period announced a partnership with Cerebras Systems, integrating that company’s inference spot connected servers moving Trainium for what Amazon promises volition beryllium superpowered, low-latency AI performance. 

But Amazon’s ambitions spell beyond the chips themselves. It besides designs the server that hosts the chips. Besides the networking components, this squad has designed “Nitro,” a hardware-software combo that provides virtualization tech (which allows galore instances of bundle to tally separately connected the aforesaid server); caller state-of-the-art liquid cooling technology; and the server sleds (pictured below) that big this gear. 

All of that is to power outgo and performance. 

AWS Austin spot laboratory tour, sled with components.Image Credits:TechCrunch/Julie Bort

Working 24/7 connected the “bring-up” 

Amazon’s customized chip-designing portion was calved erstwhile the unreality elephantine bought Israeli spot decorator Annapurna Labs successful January 2015 for astir $350 million. So this squad has present had much than 10 years designing chips for AWS. The portion has retained its Annapurna roots and sanction — its logo is everyplace successful the office. 

This spot laboratory is located successful a shiny, chrome-windowed gathering successful Austin’s upscale “The Domain” district, a walkable country filled with shops and restaurants that’s sometimes called Austin’s Silicon Valley. 

The offices person your classical tech firm vibe: desks successful cubicles, gathering spots, and league rooms. But tucked distant astatine the backmost of a precocious level successful the gathering is the existent lab, with sweeping views of the city.  

The shelving-filled lab, astir the size of 2 ample league rooms, is simply a noisy concern abstraction acknowledgment to the fans connected the equipment. It looks similar a transverse betwixt a precocious schoolhouse store people and a Hollywood acceptable for a high-end lab, but the engineers are dressed successful jeans, not achromatic laboratory coats.

AWS Austin Chip Lab.Image Credits:TechCrunch/Julie Bort

AWS Austin spot lab.Image Credits:TechCrunch/Julie Bort

Note that this is not wherever the chips are manufactured, truthful nary achromatic hazmat suits were necessary. The Trainium3 is simply a state-of-the-art 3-nanometer chip, produced by TSMC, arguably the person successful 3-nanometer manufacturing, with different chips produced by Marvell. 

But this is the country wherever the magic of the “bring-up” occurs.  

“A silicon bring-up is erstwhile you get the spot for the archetypal time, and it’s similar a large overnight party. You enactment here, similar a lock-in,” King explains. After 18 months of work, the spot is activated for the archetypal clip to verify it works arsenic designed. The squad adjacent filmed immoderate of the Trainium3 bring-up and posted it connected YouTube.

Spoiler alert: It’s ne'er problem-free.  

For Trainium3, the prototype spot was primitively air-cooled, similar erstwhile versions. The existent spot is present liquid-cooled, which offers vigor advantages and was rather an engineering feat.

During the bring-up, the dimensions for however the spot attached to the air-cooling vigor descend were off, truthful the spot couldn’t beryllium activated. 

Unfazed, the squad “immediately got a grinder and conscionable started grinding disconnected the metal,” King said. Because they didn’t privation the sound disrupting the bring-up pizza enactment atmosphere, they snuck disconnected and did the grinding successful a league room.  

Staying up each nighttime and solving problems “is what silicon bring-up is each about,” King said. 

The laboratory adjacent has a welding station, wherever hardware laboratory technologist and maestro welder Isaac Guevara demonstrated welding tiny integrated circuit components done a microscope. This is specified insanely hard enactment that elder person Carroll openly admitted helium couldn’t bash it, to the guffaws of Guevara and the remainder of the engineers successful the room. 

AWS Austin spot laboratory tour, welding station.Image Credits:TechCrunch/Julie Bort

The laboratory besides contains some custom-made and commercialized tools for investigating and analyzing issues with chips. Here’s awesome technologist Arvind Srinivasan demonstrating however the laboratory tests each tiny constituent connected the chip:

AWS Austin spot laboratory tour, investigating equipment.Image Credits:TechCrunch/Julie Bort

Sleds are the prima of the lab 

But the prima of the laboratory is an full enactment showcasing each procreation of the “sleds” the squad designed. 

AWS Austin spot laboratory circuit partition of sleds.Image Credits:TechCrunch/Julie Bort

Sleds are the trays that location the Trainium AI chips, Graviton CPU chips, and supporting boards and components. Stack them unneurotic connected a rack with the networking component, besides custom-designed by this team, and you get the systems that are astatine the bosom of Anthropic Claude’s success. 

Here’s the sled that was shown disconnected during the AWS re:invent league successful December: 

AWS Austin spot laboratory tour, Trainium3 sled.Image Credits:TechCrunch/Julie Bort

Proven by Anthropic and OpenAI

I expected my guides to crow astir the OpenAI woody during the tour. But they didn’t. 

The reticence could person been related to the aforementioned imaginable ineligible haze that mightiness bent implicit the deal. But the consciousness I got was that these boots-on-the-ground engineers (who are presently designing the adjacent version, Trainium4) haven’t had overmuch accidental to enactment with OpenAI yet. Their day-to-day enactment has truthful acold been focused connected Anthropic’s and Amazon’s needs.

Currently, the biggest chunk of Trainium2 chips is deployed successful Project Rainier — 1 of the world’s largest AI compute clusters — which went unrecorded successful precocious 2025 with 500,000 chips. It’s utilized by Anthropic. 

But determination was a partition show successful the main bureau displaying a punctuation astir however OpenAI volition beryllium using Trainium. The pridefulness was there, if subtle.  

In summation to this lab, the squad besides has its ain backstage information halfway for prime and investigating purposes. A abbreviated thrust away, it doesn’t tally lawsuit workloads, truthful it’s housed astatine a co-location facility, not an AWS information center.

Security is tight: There are strict protocols to participate the gathering and to entree Amazon’s country within.

The information center’s cooling strategy is truthful large that earplugs are mandatory, and the aerial is heavy with the acrid odor of heated metal. It’s not a pleasant spot for the mean idiosyncratic to bent out. 

Here’s maine and Aronson astatine the AWS Austin spot laboratory information center, protecting our ears adjacent to unrecorded servers.Image Credits:TechCrunch / Julie Bort

At this information center, determination are rows and rows of servers filled with sleds that integrate each of Amazon’s newest customized chips: Graviton CPU, liquid-cooled Trainium3, Amazon Nitro, each happily computing away. The liquid runs connected a closed system, meaning it is reused, which should besides assistance trim the biology impact, the engineers said. 

Here’s what a existent Trn3 UltraServer looks like: Multiple sleds are connected apical and bottom, with the Neuron switches successful the middle. Hardware improvement technologist David Martinez-Darrow is seen present performing attraction connected a sled:

AWS Austin spot laboratory circuit information center.Image Credits:TechCrunch/Julie Bort

While attraction connected the squad has ever been high, the scrutiny has truly ratcheted up arsenic of late. 

Amazon CEO Andy Jassy keeps a adjacent oculus connected this lab, publically bragging astir its products similar a arrogant dad. In December, helium said Trainium was already a multibillion-dollar concern for AWS and called it 1 portion of AWS tech he’s astir excited about. He besides gave the spot a shout-out erstwhile announcing the OpenAI agreement.  

The squad feels the pressure, too. Engineers volition enactment 24/7 for 3 to 4 weeks astir each bring-up lawsuit to hole immoderate issues truthful the chips tin beryllium mass-produced and enactment into information centers.

“It’s precise important that we get arsenic accelerated arsenic imaginable to beryllium that it’s really going to work,” Carroll said. “So far, we’ve been doing truly well.” 

*Disclosure: Amazon provided airfare and covered the outgo of 1 nighttime astatine a section hotel. Honoring its Leadership Principle of Frugality, this was a back-of-the-plane mediate spot and a humble room. TechCrunch picked up the different associated question costs similar Ubers and luggage fees. (Yes, I checked a container for an overnight trip. I’m precocious attraction that way.)