Solera Networks - See Everything. Know Everything.

The Speed Revolution

Alan Hall - Director of Marketing — Fri, 29 Jan 2010 00:09:37 +0000

While coffeehouses in Europe have been around since the 1500s, the coffee pot was invented in 1806 with a metal sieve to strain away the grounds.

It took another 166 years for the Mr. Coffee® brand to invent the first automatic drip coffeemaker and its disposable filter that provided convenience, affordability and a reason for every household to stock up on Folgers.

That was a year after the first Starbucks coffeehouse opened its doors in 1971.

And the rest is history. Starbucks is now on every corner and it is hard to imagine a world without a triple grande sugar free vanilla latte.

The revolution can be seen just as clearly when we look at the world of PCs. But, not many industries have changed as quickly or dramatically. Just look at the past 20 years—computers used to take up an enormous amount of space. But, today netbooks and Apples new iPad are smaller than a spiral-bound notebook. What will the next 20 years bring? The race for faster, smaller computers with unthinkable amounts of memory will definitely continue with fierce competition targeting organizations and consumers alike.

One way computers have changed dramatically is … speed. The “father of supercomputing” Seymour Cray created a supercomputer in 1964 that performed up to three million instructions per second, a processing speed that was three times faster than that of its closest competitor, IBM.

By today’s standards, that is an extremely slow computer network. So, as phones get smarter, computer networks get faster and companies stretch beyond borders, what does it mean for network security?

To keep up with the vast amount of data and information on a company’s network, many IT departments are working to upgrade their 1G networks to 10G and beyond. Today, security professionals are challenged to keep pace with the increasing speed of their networks, especially when it comes to finding the source of an attack when such vast amounts of data are crossing the network at such speeds. The ability to capture, index and store all of the traffic on a full 10G network is extremely important for companies who have moved beyond just trying to prevent a security breach.

Yet, most organizations don’t realize that they can indeed capture, record, index, search, and retrieve any network data at ultra-fast 10Gb network speeds. They still feel restricted by slow technologies that take hours, days, even weeks to retrieve recorded data. The result? Many have given up and consigned themselves to just looking at metadata and hoping they’re protected by their signature-based security tools. They are not recording their network traffic 24/7/365 because they still think it is an impossible task. So when attacks like “Operation Aurora” occur, they have little knowledge of whether their network has been a target, and the extent of the attack.

It is time for organizations to realize that network forensics solutions have been revolutionized to keep up with the speed of today’s networks. It is not a vision of the future—It is here and now.

The Solera DS Appliance captures a company’s comprehensive stream of network traffic, full packet header and payload, indexes those packets into easy-to-search and access flows for replay and artifact reconstruction. So what’s the advantage of speed? Well, you can only analyze and uncover what you have recorded. If you didn’t record it, you don’t have the evidence. Active network forensics that can accommodate today’s network speeds enables real-time analysis without any network impact. It is vital to simply replay a suspected security incident within seconds or minutes, not days or weeks, to quickly identify the source and scope of the attack. To our customers, speed matters. Speeds that today seem beyond comprehension to some will be the norm of the future—and it will be here faster than you think.

So, what’s the best part of waking up? It’s not Folgers, It’s knowing that when your company is the target of an attack, you will be able to swiftly identify and remediate the breach, and protect the organization from further exposure – all prior to your morning coffee break.

Operation Aurora: Prepare for Cyberwar

Steve Shillingford - CEO — Fri, 22 Jan 2010 12:00:39 +0000

Two weeks ago when Google announced they were the target of sophisticated attacks from China, we were notified a war had begun. China had a plan to attack and steal Google intellectual property and compromise Gmail. Google was not alone. Coordinated attacks were also organized targeting Adobe and nearly 20 other corporate and government sites.

The attacks, known now as “Operation Aurora” took advantage of a Microsoft Internet Explorer vulnerability. Today, Microsoft announced a patch for this particular vulnerability.

What does this latest attack tell us? Quite simply, you cannot fight a global cyber war without sufficient weaponry. Would we expect our military to enter into Iraq with just knives? Absolutely not. Similarly, we cannot expect our flagship brands in American enterprises and our government to face cyberwar without the proper tools and ability to respond.

What happened to Google, Adobe and others can never be completely prevented, but the extent of the attack could have been minimized. With active network forensics solutions in place at appropriate points in the network, these organizations could have instantly investigated all the network traffic and swiftly identified suspicious activity at the first sign of an attack. This recorded data could have been replayed to determine the exact scope and extent of the attack, including compromised systems and data. This record could have also proved what systems were not compromised, allowing these organizations to effectively remediate and protect against further exposure. With active network forensics, network traffic and information could have been retrieved in seconds, reducing the exposure window from weeks to hours.

Today, the update code for the patch is available, but what if something got in while the door was open? You may have closed it with a patch, but what about the time between exposure and patching?

It is unwise for organizations today to rely on prevention tools alone and assume they are prepared for an attack. Being able to record your traffic, review attack information and immediately respond to an enemy is an absolute must. Today, every CSO and security administrator must realize that without measures to instantly remediate an attack, they are in jeopardy. Operation Aurora has taught us a very necessary lesson indeed. After all, who brings a knife to a gunfight?

SEKE Podcast – Introduction to Solera DeepSee

Matt Wood - Cheif Scientist — Wed, 20 Jan 2010 20:39:10 +0000

Solera DeepSee Forensics Suite: Where web searching meets active network forensics.

A conversation between James HIlliard, host of the See Everything, Know Everything podcast, and Matthew Wood, Solera Networks’ Chief Scientist.

“DeepSee is a tool that provides administrators, users and even C-level people at a corporation a way to see what happened on a network as it is happening or in the past without needing to know what a packet is or what a MAC address is or how IP addresses work.”

A More Holistic Approach to Security

Pete Schlampp - VP of Marketing and Product Management — Mon, 18 Jan 2010 18:15:48 +0000

As we look back on 2009, we shouldn’t be surprised about the many “surprise” security incidents that came to light. It didn’t really matter how prepared an organization was or how compliant they were to industry standards—hackers still found a way to access the organization’s network and data and commit a crime.

In 2010, we will undoubtedly face new threats. The question is, will security professionals (look in the mirror) change their myopic focus on prevention and apply some serious attention to preparing for the unexpected threat? We need to address all three pillars of an effective security strategy: prevention, detection and incident response.

To security professionals who have prescribed to a “prevention is good enough” belief, preparedness might seem like an impossible task. But, how can we anticipate each and every threat that is out there? To help with a New Year’s resolution to be better prepared in 2010, I’d like to highlight a few steps to move toward a more holistic approach to security:

Be prepared. It’s not all about prevention. As security professionals, we can’t focus completely on prevention, because we can only stop what we know. The “unknown unknowns” will continue to roam in the wild and until they are identified and classified, any amount of prevention is insufficient. These threats will be targeting vulnerabilities we are not aware of. Just look at the vast number of recent security incidents, including T-Mobile, Heartland Payment Systems, TJX Companies, MasterCard, American Express and many others. Eventually, vulnerabilities will be found and exploited and a breach will occur. We need to take steps to prepare for the “unknown unknowns” by planning for swift, intelligent response. Incident Response needs to evolve into Instant Response.
Furthermore, we can’t put all our trust in vendors who convince us that they can stop every security incident. Simply put, it cannot be done. If you put all of your security dollars toward prevention, will you be successful? No. Come to grips with the fact that vulnerabilities exist and be prepared for the incident when it occurs.

Don’t rely on compliance alone. Compliance is a start, but regulations are really there just to provide a framework—and force adherence to—good security practices. These are practices that serious security professionals are already following. For those of us that believe that we will not be hacked because we are complaint with industry standards, think again. It can and does happen.
The Heartland breach is a great example and it has taught us that compliance alone is not enough to stop a major security breach. While Heartland was compliant with the requirements of the Payment Card Industry Data Security Standard (PCI DSS), the company still experienced the biggest breach ever involving payment card data. There are simply no guarantees when motivated attackers have an eye for your assets.

Rigorous updating and patching, along with practicing general security hygiene is of course wise; however, still not sufficient. For those deploying new technologies touted as the ultimate barrier against a security breach, proceed with caution. Eventually, someone will figure out how to hack into the new technology and you won’t have any improved capability to respond to the previously unknown threat.

Investigate, detect, and fortify. Lastly, we must understand that securing our networks and data also includes swift detection of the source and scope of any security incident. This is critical to enable quick and intelligent response. Rapid detection of a breach is arguably more important than just trying to prevent one. This holistic perspective helps you know exactly what is going on within your networks. Then, when something questionable happens, swift response to mitigate the incident provides more protection to your organization’s bottom line and brand equity than with prevention alone.

Cyber Security Czar Returns to Take his Place at the White House

Alan Hall - Director of Marketing — Tue, 22 Dec 2009 23:40:12 +0000

After much anticipation, the role of cyber security czar has been filled by none other than Howard Schmidt. Schmidt returns to service after previously being in the cyber security czar role for the Bush administration. This return to service demonstrates a change to the originally proposed position of cyber security director. Schmidt will have regular access to President Obama.

Many people have turned down the offer to become our next federal cyber security director. The biggest compliant – not enough power to create real change. But today, we are hopeful. Since Schmidt has had this position before, he fully understands the challenges and most importantly, he understands the politics of government. He also served in the CSO role in the private sector including holding this position at Microsoft. Combined with his experience in forensics and computer crime, he will provide broad experience to this role. We hope this will provide unique understanding and shape his view of the challenges in related to cyber security. He will need added courage to make some significant changes that will have an impact on the industry by taking a more realistic and effective approach to security.

Our suggestion: Stop focusing all attention and resources on known threats and acknowledge susceptibility to security loopholes, or unknown threats, that facilitate failure. We hope Schmidt moves the industry to look at cyber security in a broader context and begin to see that preparedness is not only important, it’s fundamental in addressing our objective of comprehensive security.

We are hopeful that indeed we will see change in the industry based on President Obama’s appointment of Schmidt. We also offer our full support to ensure that government agencies and enterprise alike are embracing preparedness.

The Singularity Is Not All That Near

Joe Levy - CTO — Tue, 24 Nov 2009 15:00:55 +0000

By now, no one with electricity hasn’t heard about the NSA data-center that is planned for Utah. First mention of it was seen in the wild as far back as May 2009 in H.R. 2346 (”Making Supplemental Appropriations for the Fiscal Year Ending September 30, 2009, and for Other Purposes” (search for ‘Utah’). Digging just a bit deeper takes the time line back as far as April in a supplemental Whitehouse document that included Department of Defense appropriations (see page 66) for “site preparation in advance of data center facilities construction projects to be constructed at the Utah National Guard site at Camp Williams, Utah”. The same document also provides some evidence of the seriousness of our nation’s posture on cyberwarfare, as page 67 explains: “The FY 2009 Defense Appropriations Act funded a National Security Agency project in the Operation and Maintenance, Defense-wide account. It has been determined that the project is more properly funded through the Military Construction, Defense-wide account. This provision would realign these funds from Operations and Maintenance, Defense-Wide to Military Construction, Defense-Wide.”

Understandably, Utah has been abuzz about the NSA data center for some time. Performing a search for “NSA” on the Salt Lake Tribune site yields a number of variably interesting results, each shedding bits of light on the plan and its progress. The earliest piece, dating back to July 1st, does a good job rationalizing the decision to build the massive data center in Utah, opening with: “Hoping to protect its top-secret operations by decentralizing its massive computer hubs…” and later explaining that: “The NSA’s heavily automated computerized operations have for years been based at Fort Meade, Maryland, but the agency began looking to decentralize its efforts following the terrorist attacks of Sept. 11, 2001. Propelling that desire was the insatiable energy appetite of the agency’s computers. In 2006, the Baltimore Sun reported that the NSA — Baltimore Gas & Electric’s biggest customer — had maxed out the local grid and could not bring online several supercomputers it needed to expand its operations.” Environmentalists will both mourn and be fueled by the juicy tidbit from this same piece that the data center “will also require at least 65 megawatts of power — about the same amount used by every home in Salt Lake City combined.”

Curiously, an additional piece of historical information the article fails to mention as possible site-selection rationalization is that Utah was previously selected by the NSA, back in February of 2006, for the linguistic capabilities of its returned missionaries. It would not be at all surprising if this was a factor in Utah’s being the place for what’s been described as a “collection point for surveillance of domestic and international telecommunications“.
So as they say: السلام عليكم, 企鹅性骚扰, יישר כח, etc., Utah.

The piece from July 2 provides some more information on the purpose, cost, and composition of the data center: “The supercomputers in the center will be part of the NSA’s signal intelligence program, which seeks to ‘gain a decisive information advantage for the nation and our allies under all circumstances’” and “President Barack Obama last week signed a spending bill that included $181 million for preparatory construction of the Camp Williams facility and tentatively agreed to two future phases of construction that could cost $800 million each” and “About $70 million has been budgeted for security, including vehicle inspection facilities, fencing, surveillance and separate visitor control centers for construction and technical personnel.”

I can’t yet say anything about the collection of supercomputers, but the eyewitness commentary I can provide as a commuter who drives past the planned construction site everyday is that it seems they’ve already spent more than $70 million on fencing alone, and it’s mostly resulted in heaping piles of deer roadkill. Inexplicably, the rutting deer seem to excel at finding their way through the fence to get onto the road, but can’t seem (literally, to save their own lives) to find their way back through to get off the road. Perhaps experimentation on bloated stinking mangled deer is somehow part of the grand government conspiracy.

July 7 offers up two pieces. The first objectively treats the data center as little more than fiscal stimulus (construction is planned to employ 4,000 to 5,000 people), while the second seems its calculated social counterbalance, offering up the obligatorily banal “we’ll follow orders and won’t ask any pesky questions about civil rights” shtick. At least the comments prove to be far more entertaining than the articles themselves.

The piece from October 23 was the first “mainstream” report in the Tribune on the event, getting somewhat lost in the echo chamber of reports and blogs and tweets that hit at about the same time, triggered by the Office of the Director of National Intelligence press conference (video and transcript). Win-win-win!

It’s an excerpt from this piece that is among the most important of all the coverage offered, noting the crucially irreplaceable role of people in the technologically-driven field of cybersecurity, and citing a report that is recommended reading for anyone in the information security field or the Intelligence Community:

But only a very small slice of the information stored at the center in southern Salt Lake County will ever be scanned by human eyes. And that’s the reality for most of what is collected by the nation’s other spy agencies as well. In a report commissioned by the Department of Defense last year, the Jason defense advisory group warned that the millions of terabytes of data coming into U.S. spy agencies through ever-improving sensors are being wasted. … It cited Massachusetts Institute of technology defense expert Pete Rustan, who complained that “70 percent of the data we collect is falling on the floor” [because sensor data was failing to be captured and processed].
…
“We have been blessed with a lot more sensor-type capabilities,” [said George Eanes, vice president of business development at Modus Operandi, a Florida software company that serves the defense intelligence community.] “That can be a big advantage to have in the theater, but it’s just data. You still got to have the humans in the loop before you make any decisions.”

Data Analysis Challenges

The same report cited above was also recently referenced by FAS (Federation of American Scientists) through their Secrecy News project (”Through research, advocacy, and public education, the FAS Project on Government Secrecy works to challenge excessive government secrecy and to promote public oversight”) in a post on the challenges of dealing with large data sets. The December 2008 JASON (not an acronym) report titled “Data Analysis Challenges” is a must read. Seriously – read it. Notable concepts from this “study commissioned by the Department of Defense (DOD) and the Intelligence Community (IC) on the emerging challenges of data analysis in the face of increasing capability of DOD/IC sensors”:

As the amount of data captured by these sensors grows, the difficulty in storing, analyzing, and fusing the sensor data becomes increasingly significant with the challenge being further complicated by the growing ubiquity of these sensors. (page 1)

The JASON report opens by summarily describing the challenges facing the Intelligence Community as storing, analyzing and fusing the ever-increasing amounts of data. Storing the data, obviously, should be recognized as foundational to anything but the most cursory analysis, the kind of superficial examination that the report describes as “rapid time scale” (more on this later). Yet despite storage being an unmistakable prerequisite to any kind of deeper, longer time scale analysis, there are today technology vendors hawking data-analysis wares that fail to meet this basic requirement. Because they haven’t figured out how to solve the technical challenge, they attempt to dismiss their critical deficiency with one of two arguments from ignorance: either that high-speed data capture is not possible, or that it’s not necessary.

No one would disagree that in intelligence work, data analysis is more productive than raw data capture, but likewise, no one should suggest that meaningful data analysis is possible today without having all of the data to analyze. As the report states on page 3: “the notion of fully automated analysis is today at best a distant reality.” Companies making a claim that effectively amounts to “we analyze 100% of the data that we don’t fail to capture” does nothing but betray their lack of understanding of the requirements of the Intelligence Community. Best-effort approaches can make sense when coping with current real limitations of computation or storage, but only when employed sensibly; failing to store all relevant data means never being able to analyze that un-captured data, whereas failing to analyze captured data superficially in real-time still means being able to analyze it more deeply subsequent to capture.

But storage should really only be considered table stakes. The practical utility of any storage system comes from the combination of efficient capture *and* efficient retrieval. The capture of the data should be considered the relatively easy part, and the report correctly makes clear that “the main issues in managing this volume of data are not rooted in hardware but in software” (page 23). It goes on to offer an example from the Pan-STARRS project of how commodity-off-the-shelf (COTS) hardware can be used to “serve 3 Petabytes for roughly $1M”. (The Pan-STARRS “Distilling Science from Petabytes” presentation itself is cited in the report’s end notes. A web search by name will turn up a link to the presentation which is also worth a glance. It’s of service for me to note the bullet on slide 10 which advises: “Science goals require all the data to be accessible and useful: waste no photons“.) Further, the JASON report describes some of the greater challenges when dealing with these quantities of data, namely, those of managing large data sets:

These include dealing with the complexity in the name-space that is introduced by the enormous capacity of these high performance file systems, and managing the vast archives of data that are produced by both simulation and data collection systems. Old paradigms for locating data based on a simple file path name break down when the number of files exceeds 10^9 as they now frequently do. Users have expressed the desire to locate data based on other properties of the data beyond its file name, including but not limited to its contents, its type and other semantic properties. Such a location service will require new indexing techniques that are currently subjects of academic research. (page 28)

In addition to limitations of conventional filesystems, the report also describes frustrations with commercially available databases, focusing on the paradigmatic experiences of the scientific community:

Broadly speaking the segment of the scientific community that is pushing the forefront of large-data science has been disappointed with the capability and the performance of existing databases. Most projects have either resorted to partitioned smaller databases, or to a hybrid scheme where metadata are stored in the database, along with pointers to the data files. In this hybrid scheme the actual data are not stored in the database, and SQL queries are run on either the metadata or on some aggregated statistical quantities. (page 61)

The authors of the report were astute to make this connection, acknowledging in the executive summary (page 1) that “it is of value to consider the evolution of data storage requirements arising from data-intensive work in scientific fields such as high energy physics or astronomy.” This perspective strongly validates some of Solera Networks inventions in the areas of massively-scalable (DSFS), attribute-based (GaugeFS) filesystems and databases (SoleraDB) (details available under NDA); it also helps illuminate the unique value of having a Chief Scientist (Matt Wood) who is hours from completing his Theoretical Physics PhD work in the Telescope Array Physics group at the University of Utah.

As a quick exercise to appreciate the value of real solutions to the problems encountered with traditional filesystems and databases when attempting to capture and use large sets of network traffic, consider the following:

So you’ve captured just over 3 days of traffic on your generally 1/3 utilized 10Gbps network:

That’s about 100TB of data
For around 183 billion “average” sized packets (600 bytes)
At an average of 650,000 packets per second

And now you want to find all the packets from IP address 71.213.89.177:

Do you read through 50 x 2TB or 50,000 x 2GB files?
Wouldn’t it be helpful to have an index?
Which databases efficiently handle 650,000 inserts per second?

Time Scales

As mentioned earlier, there are different time scales on which data analysis can be performed. Sensitivity to different time scales is important, and the report notes this in the executive summary: “The key challenge is to empower the analyst by ensuring that results requiring rapid response are made available as quickly as possible while also insuring that more long term activities such as forensic analysis are adequately supported.” In greater detail, it broadly distinguishing three cases (page 51):

Long time scale Here there is no critical timeliness requirement and one may want to establish results on a time scale of perhaps days. Applications which match well include retrospective analysis of multiple data sources, fusing of new data to update existing models such as geographic information systems or to establish correlations among events recorded through different information gathering modalities.

Medium time scale Such a time scale corresponds to activities like online analysis with well structured data. Typically this is accomplished in an interactive way using a client-server or “pull based” approach.

Rapid time scale In this scenario, one wants to be cued immediately for the occurrence of critical events. The time scale here may be very near real time. We will argue that a “push based” or event driven architecture is appropriate here.

The long time scale section makes cloud-computing recommendations to MapReduce / Hadoop, and also makes the wise suggestion “to move computation close to the data rather than move the data to a central point where computation takes place. This minimizes congestion and is more scalable in that there are fewer load imbalance bottlenecks due to data motion or computation” (page 59). With regard to network forensics, it would reasonable to consider such tasks as cryptanalysis, steganalysis, and statistical data mining as likely long time scale candidates.

The medium time scale section recommends the use of a service oriented architecture (SOA – the fundable version of RPC) noting its attractiveness in “applications where large data stores need to interoperate and where fusion of their data is required at a higher level.” It covers IARPA’s open-source Blackbook 2 project (”a graph analytic processing platform for semantic web”) which appears to be a non-commercial alternative to the impressively scalable and extensible Palantir data analysis platform (you can get a feel for it by playing an online game they provide, or using it to work with data from data.gov). In the spirit of the JASON report’s recommendation to modularity and sharing, and consistent with Solera Network’s practice on platform collaboration, Palantir avers a “fundamental belief that this openness will lead to long-term customer success over inflexible, closed, and proprietary solutions.” Most sorts of collaborative data analysis could fit into into the medium time scale, and scalable, high-performing, intuitive platforms will make it easier for human analysts to find interesting and valuable results in the data.

The rapid time scale is also described by the report as an “event driven architecture” (EDA) where an event is “simply a significant change of state associated with some data that is being constantly monitored.” The report differentiates an EDA from an SOA by explaining

EDA applications use a publish/subscribe model where loosely coupled software components subscribe to event streams and then either react by actuating some response or by emitting subsequent events to other components. The key idea behind this approach is asynchronous broadcasting or “push” of events.

This fairly accurately describes the sort of integration that exists between Solera Networks platforms and other event generating platforms such as SonicWALL and ArcSight, where pre-classified security events are detected on a rapid time scale through DPI pattern-matching, or security information/log aggregation. Since the platforms generating these sorts of events (either directly, or indirectly, e.g. through a SIEM) are generally in-line traffic-processing devices, their classification of events must occur in real-time (i.e. with latencies imperceptible to users), and cannot today be compared to the deeper sorts of data mining analyses that are possible in medium to long time scales. That is not to say that longer time scales are better than rapid time scales, but rather that both are necessary. I am simply recognizing the difference that exists today between a necessarily fast-twitch intrusion detection/prevention system, and a necessarily more persistent data analysis platform. Rapid time scale, event driven architectures are very good at detecting and preventing reconnaissance attempts, denial of service attacks, and known-exploit attacks easily identifiable by machines, and this type of defense is essential to protecting the day-to-day operation of information systems against tools the like of those found on milw0rm or exploit-db. But it requires longer time scales and the neocortex of a human analyst to detect the unique and unpredictable actions executed by a competent and determined criminal or terrorist agent.

That’s A Very Expensive Cat

I don’t take the fact that Utah’s NSA data center is expected to include more than 1 million square feet of space staffed by only 200 people as an indication that the NSA believes computers provide more value than analysts. Instead, I see these numbers as acknowledgment of a recognized shortage of qualified analysts. Whether its a DHS initiative to hire 1,000 cybersecurity experts over the next 3 years, or a Booz Allen Hamilton study stating that “There is a radical shortage of people who can fight in cyber space—penetration testers, aggressors and vulnerability analysts… My sense is it is an order of magnitude short, a factor of 10 short” there’s no shortage of evidence that we need more human analysts. Today’s silicon and algorithms—fast and clever as they are—get ever-better at assisting humans, but they are still far from being up to the task of understanding or analyzing the behaviors and actions (particularly the pathological behaviors and actions) of other humans.

For perspective on where state-of-the-art computing is relative to the human analytic capabilities, I’ll close with one of the more interesting announcements that just came out of SC09:

Scientists, at IBM Research-Almaden, in collaboration with colleagues from Lawrence Berkeley National Lab, have performed the first near real-time cortical simulation of the brain that exceeds the scale of a cat cortex and contains 1 billion spiking neurons and 10 trillion individual learning synapses.

The simulation was performed using the cortical simulator on Lawrence Livermore National Lab’s Dawn Blue Gene/P supercomputer with 147,456 CPUs and 144 terabytes of main memory.

We need more human analysts, and they need the government, academic, and private sectors to understand their needs well enough to provide them genuinely functional, constantly evolving tools. Kurzweil (either unfortunately or fortunately) was off by a few years. We still have quite a while to go before this:

Emerging Technologies: SSI at SC’09, (and its principle mistake)

Matt Wood - Cheif Scientist — Mon, 23 Nov 2009 17:00:04 +0000

The Intel Server System Infrastructure (SSI) Project has a lofty goal: To standardize the hardware for x86/x86_64 based blade servers and their backplanes. This is an enterprise and academic computing game changer without a doubt, but its current incarnation leaves a big hole in the debugability and security for applications and operating systems running on theblades.

So, what’s a blade server?

I’ll get to exactly what a blade server is shortly, but first an analogy serves well for visualization: think of your local telephone company and the phone line they provide you. If you want an extra telephone in your house and you’ve already got the connectors wired, you buy a new phone, connect it to the jack, then you’ve instantly got a dial tone with no fuss and no pain. This is exactly how blade servers are supposed to work for the computing industry. Put simply, a blade server (or simply “blade” for short) is a modular computer; a self contained motherboard, processor, RAM, and storage module that’s easily plugged or unplugged from a rack specifically designed to accept them. It typically has a proprietary connector containing power, network, and health management functionaliy that’s automatically connected and configured when the blade is plugged into the backplane (which is the “jack”.)

So, when a company or university needs more compuational power because the web-server is bogged down generating pages or because a scientific simulation would be too slow otherwise, they buy an extra blade and plug it in to instantly get another computing resource. Of course they probably need to configure software on the new computer for it to be useful, but that’s not important for this discussion. The take-away point is that a blade gives a no-fuss method for additing additional computers to your infrastructure, and as a bonus, when a computer inevitably fails blades give a great way to swap out the failed module with a new one in a matter of seconds.

The SSI Project

A major problem hindering blade adoption has always been the
lack of any standard (blades have been around en-masse for at least a decade, and yet most haven’t ever heard of them!) Concretely, Appro will sell you their own design for a backplane and blade, which is different then the one IBM sells, which is different than the one Dell sells. Of course, this is a headache for numerous reasons. First there is inherit risk in the future cost of any blade you buy; if a vendor decides there isn’t enough margin in their blade product line and doubles their prices, in general you can’t seek a third party blade as a second source to combat the margin surfing. Next, the same vendor may decide to end-of-life the very blades that your backplane accepts, leaving you searching eBay for used parts should you ever need more blades or replacement components. And worst case, what happends if the vendor goes out of business right when you buy your first backplane and single node? It’s potentially the Edsel car of the computing industry!

Intel has done something to change all of this. Their goal is of course to sell more CPUs, and blades are a perfect way for them to do so since the marginal per-blade upgrade cost is typically much less than that of a full computer, people buy more CPUs because they can afford more blades. So, they’ve pulled together a consortium of juggernauts in the blade industry, to design a standard architecture for blade servers and their backplanes to ensure that most of the drawbacks to blade infrastructure are washed away. If and when vendors adopt the standard, you’ll be able to cross company lines for sourcing blade servers and backplanes, just like you can cross company lines for hard disks, RAM, workstations, etc. today. If they pull such a feat off, it will be a landmark event in the computing industry to say the least, an event as significant as Compaq’s upheaval of the PC market with the reverse engineering and re-implementation of IBM’s BIOS, or with AMD’s implementation of the x86 processor line in the i386 and i486 processor days. Let’s hope they do.

The Gaping Hole

Unfortunately, the SSI picture isn’t all roses today; the standards committee has inadvertently created a security and debuggability nightmare.

I’ve glossed over the networking aspects of blade computing, but further discussion is warranted, because this is the cheif problem with the current SSI implementation. The backplanes for blade servers usually have an integrated network switch of some sort, with ethernet and infiniband incarnations being the most common. The utility here is clear; by including a network switch in the backplane a single network cable can be connected to the backplane and provide outside-world connectivity for every blade in the rack. There’s not a thing wrong with the idea behind this method, but the SSI implementation lacks a way to monitor inter-node traffic which probably makes the security administrator and MPI application developer readers groan.

What’ exactly is the problem, in case you didn’t catch it? (And if you didn’t catch it don’t worry — it can be a subtle point even if you’re on the periphery of one of the above categories.) The problem is that you can’t see anything that the nodes in an SSI backplane say to one another. In effect, you can only monitor the connections from the backplane to the outside world.

Consider the following illustrative case for why the current SSI specification is currently broken: A very common architecture for an internet website running an online store is to run a single or a few web-server blades that talk directly to internet shoppers serving them images, shopping carts, and other pages, with two or three times as many database blades connected to the web-servers with current information about item availability, stock, prices, outstanding orders, etc. A typical attack vector is the following: a malicious user breaks into the web-server through a known or newly developed vulnerability over an encrypted (https) link. The user then directs the web-server to fetch credit card numbers, names, and addresses from the database server, typically through the unencrypted link between the web-server and the database server, then tunnels the information through the encrypted link back to their PC. With the SSI blade system, a network forensics or capture device would have no way of seeing the unencrypted data-leakage, since it would happen exclusively on the blade backplane. In fact, unless the database query statements are audited and/or an SSL decryptor in used to feed the forensics systems, the company under attack will probably never know. In practice, most corporations have neither an SSL decryptor nor query auditing, since both are an expensive and detail-oriented tasks and their need is normally mitigated by forensics devices snooping the un-encrypted traffic.

Another, concrete example is in order. Super-computers are typically strung together from many single computers of the same makeup, obviously a prime market for blade servers. The developers of the applications that run on super-computers typically use the Message Passing Interface (MPI) framework for making the single computers act in parallel and in lock-step as one large super-computer. MPI programming is unfortunately error prone and hard, however, which is why super-computer programmers command big salaries. To debug MPI programs the quintessential method is to capture the messages that individual computers pass one-another, and examine them for errors or other incorrect behavior. With a super-computer made of SSI blades, however, this debugging paradigm is completely unavailable. A packet capture appliance has no single point of entry, and thus doesn’t see the messages that nodes pass one-another. Instead, developers need to debug through other means, like developing a framework for dumping messages to a log on each machine, collecting them, ordering them, and analyizing them, hoping that the framework didn’t miss a critical component of the message; or perhaps they could run tcpdump on each node, and hope that the traffic is slow enough for that tool to keep up (which may sound trivial but is in fact a major problem,) though in that case they still need a way to collect coalesce the resulting PCAP files.

Of course, there are many other examples of what is lost without the ability to snoop backplane network traffic, but the idea behind the problem should at least be clear with the scenarios already presented. What’s needed then, is a fix. The SSI specification can be augmented to support a network TAP port and all of these issues vanish in a blink. I’ve personally told the SSI developers about this issue, and its now on their radar. More feedback, of course, will always help.

Conclusion

The SSI platform represents a giant leap forward for the computing industry as a whole but it introduces a major security and the debugging nightmare into environments that already have too many of those things. A simple change can make the collective lives of every SSI blade user simpler, so they can worry about everything else.

Cyber War – Starting with a Blackout, Not a Boom

Pete Schlampp - VP of Marketing and Product Management — Mon, 16 Nov 2009 18:42:55 +0000

Those of us who work in cyber security understand what the future could hold for cyber warfare. We see how technology, like any asset, in the wrong hands can be used for inappropriate, even evil, purposes.

Recently, 60 Minutes aired a cyber security broadcast “Sabotaging the System.” While the story might seem sensationalized to some (they are after ratings), make no mistake the next big war may just begin with a blackout, not a boom.

President Obama has made cyber war defense a top national priority because cyber threats are one of the most serious economic and national security challenges we face as a nation. Every major defense agency, including the Departments of Defense, State, Commerce, Energy and NASA, has been infiltrated.

Jim Lewis, director of the Center for Strategic and International Studies, told 60 Minutes: “They can disrupt critical infrastructure, wipe databases. We know they can rob banks. So, it’s a much bigger and more serious threat.

“In 2007 we probably had our electronic Pearl Harbor. It was an espionage Pearl Harbor. Some unknown foreign power, and honestly, we don’t know who it is, broke into the Department of Defense, to the Department of State, the Department of Commerce, probably the Department of Energy, probably NASA. They broke into all of the high tech agencies, all of the military agencies, and downloaded terabytes of information.”

As I watched the 60 Minutes episode, I wasn’t at all surprised that more than two years later, the nation’s security experts are still uncertain and will probably never know who hacked these systems. This is why we at Solera Networks feel so strongly about the importance of active network forensics and dynamic defense.

If these agencies had comprehensive incident response plans and appropriate technology in place, they would have the capability to “rewind the tape” and likely identify what happened to specific files, data, etc., and take immediate steps to rectify the situation.

After this amount of time, not knowing how our nation’s assets were attacked, and the full scope of the attack, in my mind, is simply unacceptable. How are we going to stop the same person or group from compromising our nation’s security in the future if we have no idea who they are or how they attacked us.

With the nation’s security at risk, what is an appropriate response time? We can’t be comfortable with days, weeks, months…or never. We must settle for nothing less than swift, intelligent response.

Succeeding at Failure

Steve Shillingford - CEO — Tue, 10 Nov 2009 21:02:51 +0000

An interesting report recently came out from a government analyst firm, Input, on U.S. government spending. According to the report, spending on cybersecurity will grow at a compound rate of 8.1 percent a year between 2009 and 2014, outpacing general IT spending.

While we can debate whether that is the right number, clearly, government needs to be focusing more on protecting our most critical assets and addressing the current security holes that are in the networks. However, I was a bit confused to see that Input predicts that most of this spending will be on preventative measures. The notion of building a higher wall here comes to mind.

I read the following in an IDG article and I wanted to pound my head on the desk:

“You need to be paying attention to where those consolidation and centralization centers are,” he said. (Kevin Plexico, Input’s senior vice president of research and analysis.)

In addition, much of the federal government’s cybersecurity focus will be real-time monitoring and control of computer networks, Plexico predicted. He sees less emphasis in the future on audits to identify breaches after they happen.

“Agencies are really investing in technology that helps them identify threats as soon as they happen, and even anticipate where those threats are going to come from,” he said.

This is yet another example of people not paying attention to logic. It is simply impossible to prevent every bad thing from penetrating your network. To advertise otherwise, presupposes that we can “out-guess”, “out-anticipate”, or otherwise predict how cyber-criminals plan their attacks. It seems illogical that industry pundits continue to miss this point. The fact has always been that preventative measures are necessary, but hackers will continue to evolve in sophistication faster than vendors can create new products. It explains why so many agencies have already been compromised and why we continue to hear about them despite the billions (yes, with a “b”) having already been spent. The definition of insanity…?

While most organizations implement security strategies that target prevention of a cyber attack, those same organizations fail to understand the three pillars of an effective security strategy – prevention, detection and incident response – and the importance of each.

Prevention. We know prevention is not a 100 percent guarantee. Recent security breaches at T-Mobile, Heartland Payment Systems, TJX Companies, Lexis Nexis, Twitter, Visa and MasterCard provide proof that prevention is not an absolute. We can also use common sense to reason that there are endless potential attack vectors into a network; portraying ourselves as capable of anticipating all of these, with perfect accuracy, is to say that we can become omniscient in our deployments. Does anyone really believe this?
Most organizations implement many products and services to prevent a security incident. Network intrusion applications, application firewalls, unified threat management, data and information leak prevention, antivirus software, content monitoring and filtering, etc., are all terms well known by your IT security staff. But, what happens when a hacker is successful at breaking through your “secure” system? Read the paper this week (or any week) and I’ll bet you can find an example.
Detection. When a breach occurs (because we all know it is going to happen), what happens next? The ability to address security incidents as they happen is critical strategy organizations fail to implement, even though the cost of failure is so great.
A recent report found that the average cost to remediate a compromised health care record is $211. Here’s a costly example, the University of North Carolina recently disclosed a breach of 160,000 records. Based on $211 per record, the cost to remediate would be nearly $34 million. Interestingly, the cost to remediate the TJX breach was estimated between $250 and $300 million, based on $3 per compromised record.
Network forensics/incident response. The third pillar is quite simple: When something bad happens (and we know it will and we know that we can’t always anticipate what it will be), we plan our reaction to it. With a comprehensive incident response plan and the right technology, you simply rewind the tape. Like a bank that has just been robbed (because despite the guards, alarms, and locks, they still deploy cameras), network forensics provides organizations a rewind feature to quickly identify the full scope of any security event, determine what happened to specific files, data, etc., and then take immediate steps to resolve the situation. Reduced exposure, evolving security measures, and ultimately having a way to review an attack and make sure it can’t happen again is just good practice. It’s the same reason law enforcement agencies have forensic teams to conduct criminal investigations and New York City has arguably one of the most comprehensive incident response teams for terrorist attacks. Why do we think network security is any different from security in the physical world?

We recently conducted a survey of our own of 200 security and IT professionals in large organizations which found an overwhelming number of companies have recently experienced or expect to experience a significant network security breach within the next 36 months.

Most of these respondents expected to be attacked at some point and many said it took between 2 to 10 days or more to determine the scope of the attack (if they could determine it at all). This is telling and directly contradicts the report from Input. Instead of focusing exclusively on building higher walls and better locks, I would argue that our industry can take a page out of the real (physical) world and start to think differently about how we battle today’s most pressing national security threat.

National Breach Notification Laws

Joe Levy - CTO — Sun, 08 Nov 2009 19:08:51 +0000

As a follow-up to a post from February 2009, I’m mostly happy to comment on the recent progress that’s been made toward the establishment of National breach notification laws. As reported on November 5, 2009 by GovInfoSecurity.com, “the Senate Judiciary Committee Thursday approved two companion bills that would require businesses and government agencies to notify individuals of security breaches involving sensitive personally identifiable information. Both bills go to the Senate for consideration.”

The first, S.139 “Data Breach Notification Act”, is a short and fairly high-level bill “to require Federal agencies, and persons engaged in interstate commerce, in possession of data containing sensitive personally identifiable information, to disclose any breach of such information.” Strangely, while a bill titled “Data Breach Notification Act” would seem to be a generalized proposal for full disclosure and transparency in the event of a data breach, rather than a specific protect individuals against identity-theft measure, S.139 focuses almost neurotically on personally identifiable information. The Definitions section reasonably describes “Sensitive Personally Identifiable Information” (PII) as the usual set of some combination of name, social security #, passport #, address, birth date, biometric data, or account information. Puzzlingly, however, it perfunctorily defines “Security Breach” as:

(6) SECURITY BREACH
(A) IN GENERAL- The term ‘security breach’ means compromise of the security, confidentiality, or integrity of computerized data through misrepresentation or actions that result in, or there is a reasonable basis to conclude has resulted in, acquisition of or access to sensitive personally identifiable information that is unauthorized or in excess of authorization.

The second, S.1490 “Personal Data Privacy and Security Act of 2009″ is a toothier and far more detailed proposal “to prevent and mitigate identity theft, to ensure privacy, to provide notice of security breaches, and to enhance criminal penalties, law enforcement assistance, and other protections against security breaches, fraudulent access, and misuse of personally identifiable information.” Title I introduces penalties both for the perpetration of identity-theft crimes, and also for the intentional concealment of data breaches. Title II sets transparency requirements and enforcement for data brokers. Title III and its subtitles define the requirements and enforcements for a Personal Data Privacy and Security Program and security breach notifications, and establishes within the FTC the Office of Federal Identity Protection to help victims of identity theft. Finally, Title IV sets compliance standards for awarding contracts to data brokers, requires Federal agencies to complete privacy impact assessments before obtaining from data brokers any PII on US citizens, and amends the duties and responsibilities of the Chief Privacy Officer, reporting to the Deputy Attorney General.

Why the mixed feelings? The good: These bills offer a single national standard rather than a mélange (or sometimes completely nonexistent) state data breach laws, they seems to take the stance of “expenses be damned, we’re going to start doing the right things,” and they establish some pretty stiff enforcements and penalties. The bad (this is going to take a bit longer): First, S.139 greatly neuters the potential effectiveness of a national law by limiting itself to a delineated bag containing only personally identifiable information. What about breaches involving such losses as corporate information whose disclosure might be of interest to shareholders, or client-attorney data, or redacted medical records, etc.? Was this confinement really necessary considering the single-minded focus S.1490 has on identity-theft?

Second, the “Exemptions” sections in both S.139 and S.1490 both basically say parties are exempt from the notification requirements if they have encrypted the data or otherwise rendered the data indecipherable. Makes perfect sense given that we also accept that encryption is unbreakable, and that the ultimate utility of stolen data is something that can be assessed prior to the occurrence of a data breach.

Third, and most importantly, the surprisingly prescriptive Section 302 of S.1490 does well enough with some conventionally safe and wise words about risk assessment, training, vulnerability testing, the iterative nature of security, and a nod to the great and powerful cloud, but it falls short in the area of risk management and control. Section 302 4B basically says “control access”, “detect breaches”, “protect data at rest, in use, and in transit”, “employ data destruction”, “trace access to records”, and “ensure access entitlement”.

So what’s the failing? That this is a bill concerned primarily with breach notification—essentially a prescription for what should be done when security controls fail—but its “risk management” section is single-mindedly and conceitedly preventative. Rather than offering guidance for being better able to “determine the scope of a breach,” it basically says “don’t have a breach”. The “trace access to records” entry is the only bit that comes close to forensics, but myopically perpetuates the unfortunate industry fallacy that such information as netflows and access logs are sufficient for this task. When will we acknowledge that flows only show that a communication session took place, not what was communicated, or that logs are good at recording access that goes through conventional channels, but not so good at recording unsanctioned access that was intentionally subversive or exploitative?

Despite the obligatory criticisms, these bills are steps in the right direction. Both are good signs that our political leadership seems to be on the right track in the pursuit of information security.