Firms Rush to Fix Vulnerability in Key Internet Hardware

July 17, 2003
By Brian Krebs

Computer security experts scrambled this week to patch a potentially catastrophic vulnerability in the computer hardware responsible for directing the majority of the world's Internet traffic.

Network administrators worldwide were quietly alerted earlier this week by San Jose, Calif.-based based Cisco Systems. The company warned that nearly all of its Internet router models are vulnerable to a cyberattack that could leave them unable to process Web traffic. The alert was officially made public today.

The company issued a software fix for the problem, and added that it had no evidence that hackers had figured out how to exploit the security hole.

Cisco's routers process and direct the largest slice of the world's Internet traffic. As of the first quarter of 2003, the company had an 82 percent worldwide share of the market for Internet routers, down from 85 percent in 2002, according to the market research firm Dell'Oro Group.

With Cisco hardware so ubiquitous, experts stressed the significance of the company's security warning.

"The main thing that is really saving everything right now is nobody really knows what those exploits are and Cisco certainly isn't going to disclose that," said Oliver Friedrichs, senior manager with Symantec Security Response. "If someone were to write a worm or virus to exploit this, it certainly could be a significant threat in terms of taking networks down," he said.

Dan Ingevalson, engineering director at the research arm of Atlanta-based Internet Security Systems, agreed, calling the security flaw the "single biggest hole" in Cisco products to date.

"This is the sort of thing that could potentially take down the Internet," Ingevalson said.

Routers are devices that forward Internet traffic between networks. When users send requests for Web pages, data or e-mail out to the Internet, routers are responsible for directing that information from the user's network to the destination, and back again.

The vulnerability revealed by Cisco this week involves a flaw in the way its routers' operating system processes information. Armed with a special sequence of malformed data, an attacker could theoretically fool the router into believing it was unable to handle any more Internet traffic, effectively causing it to crash. Routers disabled by such an attack would have to be reset manually, a potentially daunting task given the widespread deployment of the devices.

"Cisco is part of the fabric of the entire Internet," Ingevalson said. "If you were to just send these packets out over the Internet, chances are a Cisco router would handle them at some stage. All the router has to do is see those packets and effectively the machine will shut down."

Cisco's warning was a textbook example of how the Bush administration would like the technology industry to handle security incidents. A key pillar of the administration's cybersecurity strategy, released in February, calls for greater and swifter information sharing between the government and the private sector.

Cisco notified the Homeland Security Department before making its announcement, giving the federal government a modest head start in patching federal agency networks.

Since that time, the department has instituted a tracking system to keep tabs on federal agencies as they work their way through the patching process, said David Wray, spokesman for the department's Information Assurance and Infrastructure Protection division.

"We've instituted a means for us to know when federal agencies have downloaded the upgrade and made the installation," he said.

Wray declined to elaborate on how far along the agencies are in mending their systems.

Cisco also notified nearly all of the major Internet service providers and strongly urged them to patch their systems. Ingevalson said judging from the level of activity at some of the nation's largest ISPs, most network providers appeared to grasp the seriousness of the problem.

"Effectively, large portions of the Internet have been rebooting over the past couple of days," he said.

Cisco spokesman Jim Brady said the company was working overtime to spread the word to all of its affected customers.

Yet, if recent history is any indicator, Ingevalson said, many systems running Cisco routers will likely remain unpatched for months to come, leaving significant portions of the Internet vulnerable to wily hackers.

"There is a large chance that a fair portion of Cisco routers won't get patched, and there are easily hundreds of thousands of devices that are vulnerable," he said.

In January, widespread failure to apply a patch allowed the "Slammer" worm to rapidly infect hundreds of thousands of systems running Microsoft's SQL server. The fast moving virus generated so much Internet traffic in such a short time that it knocked whole systems offline -- including automated teller machines and emergency response networks. Microsoft released a software patch to fix the vulnerability exploited by the worm six months earlier.

http://www.washingtonpost.com/ac2/wp-dyn/A7072-2003Jul17?language=printer