Nvidia rivals focus on building a different kind of chip to power AI products

SANTA CLARA, Calif. (AP) — Building the current crop of artificial intelligence chatbots has relied on specialized computer chips pioneered by Nvidia, which dominates the market and made itself the poster child of the AI boom.

But the same qualities that make those graphics processor chips, or GPUs, so effective at creating powerful AI systems from scratch make them less efficient at putting AI products to work.

That's opened up the AI chip industry to rivals who think they can compete with Nvidia in selling so-called AI inference chips that are more attuned to the day-to-day running of AI tools and designed to reduce some of the huge computing costs of generative AI.

“These companies are seeing opportunity for that kind of specialized hardware,” said Jacob Feldgoise, an analyst at Georgetown University's Center for Security and Emerging Technology. “The broader the adoption of these models, the more compute will be needed for inference and the more demand there will be for inference chips.”

It takes a lot of computing power to make an AI chatbot. It starts with a process called training or pretraining — the “P” in ChatGPT — that involves AI systems “learning” from the patterns of huge troves of data. GPUs are good at doing that work because they can run many calculations at a time on a network of devices in communication with each other.

However, once trained, a generative AI tool still needs chips to do the work — such as when you ask a chatbot to compose a document or generate an image. That's where inferencing comes in. A trained AI model must take in new information and make inferences from what it already knows to produce a response.

GPUs can do that work, too. But it can be a bit like taking a sledgehammer to crack a nut.

“With training, you’re doing a lot heavier, a lot more work. With inferencing, that’s a lighter weight,” said Forrester analyst Alvin Nguyen.

That's led startups like Cerebras, Groq and d-Matrix as well as Nvidia's traditional chipmaking rivals — such as AMD and Intel — to pitch more inference-friendly chips as Nvidia focuses on meeting the huge demand from bigger tech companies for its higher-end hardware.

D-Matrix, which is launching its first product this week, was founded in 2019 — a bit late to the AI chip game, as CEO Sid Sheth explained during a recent interview at the company’s headquarters in Santa Clara, California, the same Silicon Valley city that's also home to AMD, Intel and Nvidia.

“There were already 100-plus companies. So when we went out there, the first reaction we got was ‘you’re too late,’” he said. The pandemic's arrival six months later didn't help as the tech industry pivoted to a focus on software to serve remote work.

Now, however, Sheth sees a big market in AI inferencing, comparing that later stage of machine learning to how human beings apply the knowledge they acquired in school.

“We spent the first 20 years of our lives going to school, educating ourselves. That’s training, right?” he said. “And then the next 40 years of your life, you kind of go out there and apply that knowledge — and then you get rewarded for being efficient.”

The product, called Corsair, consists of two chips with four chiplets each, made by Taiwan Semiconductor Manufacturing Company — the same manufacturer of most of Nvidia's chips — and packaged together in a way that helps to keep them cool.

The chips are designed in Santa Clara, assembled in Taiwan and then tested back in California. Testing is a long process and can take six months — if anything is off, it can be sent back to Taiwan.

D-Matrix workers were doing final testing on the chips during a recent visit to a laboratory with blue metal desks covered with cables, motherboards and computers, with a cold server room next door.

While tech giants like Amazon, Google, Meta and Microsoft have been gobbling up the supply of costly GPUs in a race to outdo each other in AI development, makers of AI inference chips are aiming for a broader clientele.

Forrester's Nguyen said that could include Fortune 500 companies that want to make use of new generative AI technology without having to build their own AI infrastructure. Sheth said he expects a strong interest in AI video generation.

“The dream of AI for a lot of these enterprise companies is you can use your own enterprise data,” Nguyen said. “Buying (AI inference chips) should be cheaper than buying the ultimate GPUs from Nvidia and others. But I think there’s going to be a learning curve in terms of integrating it.”

Feldgoise said that, unlike training-focused chips, AI inference work prioritizes how fast a person will get a chatbot's response.

He said another whole set of companies is developing AI hardware for inference that can run not just in big data centers but locally on desktop computers, laptops and phones.

Better-designed chips could bring down the huge costs of running AI to businesses. That could also affect the environmental and energy costs for everyone else.

Sheth says the big concern right now is, “are we going to burn the planet down in our quest for what people call AGI — human-like intelligence?”

It’s still fuzzy when AI might get to the point of artificial general intelligence — predictions range from a few years to decades. But, Sheth notes, only a handful of tech giants are on that quest.

“But then what about the rest?” he said. “They cannot be put on the same path.”

The other set of companies don’t want to use very large AI models — it’s too costly and uses too much energy.

“I don’t know if people truly, really appreciate that inference is actually really going to be a much bigger opportunity than training. I don’t think they appreciate that. It’s still training that is really grabbing all the headlines,” Sheth said.

A d-Matrix Corsair PCIe card is shown in a server at the d-Matrix office in Santa Clara, Calif., Wednesday, Oct. 16, 2024. (AP Photo/Jeff Chiu)

A countdown clock for the release of Aviator software 2025 on Corsair to customers is displayed at the d-Matrix office in Santa Clara, Calif., Wednesday, Oct. 16, 2024. (AP Photo/Jeff Chiu)

Sid Sheth, CEO and Co-founder of d-Matrix, holds up a d-Matrix Corsair chip during an interview in Santa Clara, Calif., Wednesday, Oct. 16, 2024. (AP Photo/Jeff Chiu)

A sign reading "create" is displayed outside of a lab at the d-Matrix office in Santa Clara, Calif., Wednesday, Oct. 16, 2024. (AP Photo/Jeff Chiu)

A display of d-Matrix Corsair chips and a package of 4 chips are shown in Santa Clara, Calif., Wednesday, Oct. 16, 2024. (AP Photo/Jeff Chiu)

Sid Sheth, CEO and Co-founder of d-Matrix, is interviewed in Santa Clara, Calif., Wednesday, Oct. 16, 2024. (AP Photo/Jeff Chiu)

Sid Sheth, CEO and Co-founder of d-Matrix, poses for a portrait during an interview in Santa Clara, Calif., Wednesday, Oct. 16, 2024. (AP Photo/Jeff Chiu)

Sid Sheth, CEO and Co-founder of d-Matrix, holds up a d-Matrix Corsair chip during an interview in Santa Clara, Calif., Wednesday, Oct. 16, 2024. (AP Photo/Jeff Chiu)

A d-Matrix sign is displayed at the company's office in Santa Clara, Calif., Wednesday, Oct. 16, 2024. (AP Photo/Jeff Chiu)

Employees work behind an evaluation board, foreground, in a lab at the d-Matrix office in Santa Clara, Calif., Wednesday, Oct. 16, 2024. (AP Photo/Jeff Chiu)

SEATTLE (AP) — Northern California and the Pacific Northwest are bracing for what is expected to be a powerful storm, with heavy rain and winds set to pummel the region and potentially cause power outages and flash floods.

The Weather Prediction Center issued excessive rainfall risks beginning Tuesday and lasting through Friday as the strongest atmospheric river — long plumes of moisture stretching far over the Pacific Ocean — that California and the Pacific Northwest has seen this season bears down on the region. The storm system has intensified so quickly that it is considered a “ bomb cyclone,” explained Richard Bann, a meteorologist with the National Weather Service Weather Prediction Center.

The areas that could see particularly severe rainfall as the large plume of moisture heads toward land will likely stretch from the south of Portland, Oregon, to the north of the San Francisco area, he explained.

“Be aware of the risk of flash flooding at lower elevations and winter storms at higher elevations. This is going to be an impactful event,” he said.

In northern California, flood and high wind watches go into effect Tuesday, with up to 8 inches (20 centimeters) of rain predicted for parts of the San Francisco Bay Area, North Coast and Sacramento Valley.

A winter storm watch was issued for the northern Sierra Nevada above 3,500 feet (1,066 meters), where 15 inches (28 centimeters) of snow was possible over two days. Wind gusts could top 75 mph (120 kph) in mountain areas, forecasters said.

“Numerous flash floods, hazardous travel, power outages and tree damage can be expected as the storm reaches max intensity” on Wednesday, the Weather Prediction Center warned.

In Northern California’s Yolo County, crews spent Monday clearing culverts, sewers and drainage ditches to avoid clogs that could lead to street flooding. Mesena Pimentel said she hopes the efforts prevent a repeat of floods last February that inundated her property near Woodland.

“We had about ten inches of water in our garage, had a couple gophers swimming around,” Pimentel told KCRA-TV. Woodland city officials have set up two locations where residents can pick up free sandbags. Authorities urged people to stock up on food and charge phones and electronics in case power goes out and roads become unpassable.

Meanwhile, Southern California this week will see dry conditions amid gusty Santa Ana winds that could raise the risk of wildfires in areas where crews are still mopping up a major blaze that destroyed 240 structures. The Mountain Fire, which erupted Nov. 6 in Ventura County northwest of Los Angeles, was about 98% contained on Monday.

Winds will calm by the end of the week, when rain is possible for the greater Los Angeles area.

In southwestern Oregon near the coast, 4 to 7 inches (10 to 18 centimeters) of rain is predicted — with as much as 10 inches (25 centimeters) possible in some areas — through late Thursday night and early Friday morning, Bann said,

A high wind warning has been issued for the north and central Oregon coast beginning at 4 p.m. Tuesday with south winds from 25 mph (40 kph) to 40 mph (64 kph), with gusts to 60 mph (97 kph) expected, according to the weather service in Portland. Gusts up to 70 mph (113 kph) are possible on beaches and headlands. Widespread power outages are expected with winds capable of bringing down trees and power lines, the weather service said. Travel is also expected to be difficult.

Washington could also see strong rainfall, but likely not as bad as Oregon and California. From Monday evening through Tuesday, some of its coastal ranges could get as much as 1.5 inches (3.8 centimeters) of rain, Bann said.

The weather service warned of high winds from Tuesday afternoon until early Wednesday for coastal parts of Pacific County, in southwest Washington. With gusts potentially topping 35 mph (46 kph) — and likely faster near beaches and headlands — trees and power lines are at risk of being knocked down, the Pacific County Emergency Management Agency warned.

Washington State Patrol Trooper John Dattilo, a patrol spokesperson based in Tacoma, posted on social media Monday afternoon that people should be prepared for “some bad weather” on Tuesday night. “Stay off the roads if you can!”

A blizzard warning was issued for the majority of the Cascades in Washington, including Mount Rainier National Park, starting Tuesday afternoon, with up to a foot of snow and wind gusts up to 60 mph (97 kph), according to the weather service in Seattle. Travel across passes could be difficult if not impossible.

Outside of this region, the central and eastern Gulf Coast, including the Florida Panhandle, is at risk for flooding on Tuesday, with 2 to 3 inches (5 to 7.6 centimeters) of rainfall are in the forecast, according to the weather service. Low-lying and urban regions could see flash floods.

Weber reported from Los Angeles. Associated Press reporter Lisa Baumann contributed to this report.

FILE - A person walks along the beach with flooding along the boardwalk Thursday, Feb. 1, 2024 in Seal Beach, Calif. (AP Photo/Eric Thayer, file)