Tag Archives: Richard Bejtlich

Threat intelligence evolution

Posted on 2012-02-23 | Leave a comment

When I got started in network security many years ago, I principally dealt with assets. As time went on, I dealt more with vulnerabilities because, hey, that’s sexy. But that’s old and busted: the new hotness is threat.

Semantics: words mean things

If one thing makes me crazy about security vendors – and far more than one thing does, to tell the truth – it’s the imprecise use of language. Depending on who you ask, a piece of malware is a “risk” and unpatched software is a “threat”. Please don’t ask me what I say when an antivirus program classifies netcat as a “trojan.” Communication reflects thought, and so when you use words in fuzzy, ill-defined ways, you also think in fuzzy, ill-defined ways. So when we talk about “threats”, let’s be clear: “a threat is what we’re trying to protect against”.

I often fall into my own trap by conflating the terms “threat” and “threat actor”, usually distinguishing only via context. In reality, though, we need to understand the difference between the components of “threat”, which CERT rolls up into “an indication of a potential undesirable event”. We can break down the threat further into the actor, exploit (method), motive. Generally, threat intelligence. as most private-sector organizations use it today, centers around methods. This usually means malware indicators, network addresses, and traffic signatures. We also sometimes talk about motive: espionage, ‘hacktivism’, organized crime, and so on. These things matter. We can’t lose sight of them, but we can’t stay content with them, either.

illuminati

This would be a terrible way to deter threat actors.

With some notable exceptions, we rarely talk about intelligence based on specific threat actors. Even then, those lead to controversy because the indicators remain classified, and so we fall back on IP addresses when attribution can encompass so much more.

Crowdstrike

Today, the firm Crowdstrike announced its “stealth-mode launch”:

CrowdStrike is a security technology company focused on helping enterprises and governments protect their most sensitive intellectual property and national security information. Utilizing Big-Data technologies, CrowdStrike is developing a new and innovative approach to solving today’s most demanding cyber-security challenges. CrowdStrike’s core mission is to fundamentally change how organizations implement and manage security in their environment.

I don’t quite understand what they’ll offer, which I suppose explains the stealth mode bit. They give proper attention to the concept of attribution and TTP (tactics, techniques, and procedures), and throw out a little red meat about patriots defending against nation-state adversaries.

By identifying the adversary and revealing their unique TTPs (i.e. modus operandi), we can hit them where it counts – at the human-dependent and not easily scalable parts of their operations.

This tends to put me in mind to agree with Saso Virag, who saw three components:

That is, Crowdstrike clearly has a militia mindset, a possible focus on tracking down the humans behind the screens, and trying to find the adversary who has already penetrated the network. I don’t know whether they intend to try to conduct attacks against the attackers or simply try to defend against the non-automated portions of the kill chain. Certainly, they want to go further than sitting back while attackers simply out-maneuver defenders. The concept incurs a lot of operational risk, and I personally would quibble with some parts of it (e.g. nationalistic motivations). At the same time, though, I also agree with the general concept that the status quo can only have negative outcomes for us today, and maybe their approach will work.

Quis custodiet ipsos custodes?

Richard Bejtlich has written about this before, and he tends to lean towards striking back in various ways too. At one time, I would not have conceived that private-sector organizations could get into this role. But the trend toward private military contractors and the like over the last decade might indicate otherwise. The US has already started to outsource critical “cyber” operations to firms like General Dynamics, such as domestic monitoring of dissident elements on the Internet. The conspiracy theorist in me wants to call this “megacorp martial law”. And while the NSA and the US intelligence committee won’t confirm publicly that it has undertaken CNA/CNE, that has to be the worst-kept secret in intelligence since Israel got nuclear weapons.

So could you do this under a government contract? What about organizations based outside of the US, particularly if they do not employ US persons? On the one hand, the government might look the other way as it has with many existing “cyber militia” operators. On the other hand, when private organizations interfere with ongoing intelligence operations (e.g. bringing down a jihadist forum that the CIA has already infiltrated), then they’ll draw the attention and ire of men in dark suits with guns and badges.

I’ll look forward to seeing how we all continue to evolve in our understanding and practice of threat intelligence.

→ Leave a comment

Posted in Notes

Tagged , , , , , ,

Two Things: SIEM and DFIR edition

Posted on 2012-01-31 | Leave a comment

"Two Stick" by lucianvenutianThanks to Hacker News, I ran across the charming and thought-provoking concept of Two Things:

“You know, the Two Things. For every subject, there are really only two things you really need to know. Everything else is the application of those two things, or just not important.”

You also might think of these things as first principles, though these might represent something even more basic. After spending some time thinking about it, I came up with the following. Feel free to add your own or point out what I’ve missed.

Two things for DFIR:

  1. The bad guys always leave evidence behind.
  2. You aren’t looking for it in time.

Two things for SIEM:

  1. Log analysis matters more than log management.
  2. SIEM analysts eventually become DBAs. (Bejtlich‘s Principle)

I don’t know whether anybody else has called it that before, but I sure wish I could find the canonical reference for Bejtlich’s Principle.

→ Leave a comment

Posted in Notes

Tagged , , , , ,

Hunting trips: network traffic log analysis

Posted on 2011-12-14 | 4 Comments

Log analysis has always struck me as one of those things that gets too much superficial attention without enough attention to detail. That is, we know that we need to do it, but we don’t talk about how we need to do it. At best, we talk about making sure we collect and archive logs. Analysis plays second fiddle, even though in reality logs without analysis provide almost no value to an organization. And you’ll find greatest value in discovery of the earliest stages of an incident rather than in hindsight to understand what went wrong. Unfortunately, less than 1% of data breach investigations in the 2011 Verizon DBIR started with log analysis and review!

The analysis ideas I present below don’t even begin to represent a comprehensive view. And of course every network is different, so you will need to think about your specific needs. But this may get you thinking in directions you hadn’t previously considered. Side benefits include analysts becoming more proficient with their tools, pushing the limits and gaps in their toolset, creating baselines of their environment, and even mentoring via shared hunting trips. These could serve as foundations for SIEM use cases, but here we’re talking about active exploratory usage by an analyst.

Hunting trips in DFIR involve actively looking for possible anomalies or indications of compromise on your network. Even if you don’t find anomalies, you’ll get a better understanding of your baselines. In this post, I’ll talk about hunting through your network traffic logs. Richard Bejtlich talks about hunting through systems as well, but I’ll save that particular discussion for another day. Further, if you do this by having a junior analyst “tag along” with a more experienced analyst (e.g. via screen sharing and chatting), you get the regular benefits of good analysis plus team-building and training.

Egress traffic

First, and most importantly, always keep in mind that we’re only identifying anomalies, not automatically classifying “bad” traffic. Nothing here can positively and without question find evil with no false positives or false negatives. It should, however, increase your efficiency in finding things that violate your policies or possibly indicate a compromise.

Compromised systems may start sending out traffic that doesn’t look like the rest of your traffic. Perhaps an attacker is trying to exfiltrate data, or a bot may simply try to contact its C&C infrastructure. So look carefully at outbound traffic logs from your perimeter firewalls. Good protocol candidates include SSH, SMTP, and IRC (yes, even now in 2011). In fact, examine all non-HTTP traffic from user subnets with suspicion.

Also look for protocol-port mismatches. Do you have HTTP traffic on high ports, or maybe even something like SSH on TCP 80? Attackers often like to overload TCP 80 to slip through loosely secured perimeter networks.

Web traffic has some unique problems. Not only does it involve a constantly changing set of endpoints, protocol evolution means that HTTP isn’t really the top-level protocol in the stack anymore. Development has rapidly left behind simple GETs and PUTs, and things like WebSockets overload ports beyond what you may realize. Still, try to analyze this traffic because so much malicious activity uses this channel.

For outbound surfing, look at your User-Agent strings: lots of spyware browser extensions will show up here. Some malware tries (poorly) to look like regular browsers and you can sometimes find it through misspellings or anomalies like default languages. A good proxy may do this, but mining the data yourself can find new threats. Look at the domains that users hit as well. Check URLs against external APIs but beware. If you get the chance and it fits your network or organization, look at destination geolocation. You may identify suspicious traffic by its destination country – if you sell widgets to farmers in Iowa, then outbound traffic to Eastern Europe or the Asia-Pac region is worth a second look. For both of these areas, applying the principle of Least Frequency of Occurrence can greatly reduce the dataset you actually need to review.

Ingress traffic

Inbound traffic to your web servers should get a close look too, using similar analysis methods as we discussed for outbound web traffic. However, take a close look at your URI query strings to find people attempting SQL injection or other forms of attack (hint: look for really long payloads). You may wish to review user agents here as well, though your mileage may vary if you run a popular web site or one with lots of global exposure. This will have particular effectiveness when analyzing traffic to API servers.

Consider looking at source geolocation as well, though as before, don’t fall into traps. In some organizations, working with your marketing or web analytics team can help you understand things and clarify your assumptions here.

The effectiveness of this part of the review may vary according to your threat model and overall security posture. For example, if you don’t have a good application security program, or if you have few users on your network, this area will matter more than egress traffic. Conversely, if you have very few exposed services, this may not deserve as much effort.

Baselines

Create some network flow baselines. You can’t know what’s anomalous until you know what’s normal. A word of caution here, though: don’t assume your baselines are already secure. You might have an existing but previously-unknown compromise. So spend time with your system administrators to identify traffic flows that don’t have an immediately obvious purpose.

What does traffic in and out of your desktop networks look like? These will necessarily differ significantly from your server networks, which need the same sort of attention. What systems usually talk to each other? Do they contact a particular set of authorized external hosts (e.g. for updates and such), especially with a defined frequency? What’s the traffic distribution across various ports? Does this vary with time of day, or day of the week?

You’ll start to build a framework of known good traffic to exclude from future analyses. As the US military teaches, the more you sweat in preparation, the less you may bleed in battle.

Conclusion

Log management matters, but log analysis matters more. Even if you have a relatively limited dataset available, start with what you have. Like tugging on the proverbial sweater thread, you will find that a little effort at the beginning can quickly unravel more than you initially might have guessed.

In the future, I’ll talk about hunting trips through your systems and other types of security data. But at any time, I welcome your thoughts and suggestions!

→ 4 Comments

Posted in Notes

Tagged , , , , ,

Not a cyberwarrior

Posted on 2011-03-03 | Leave a comment

'Cyber Gang' by Anthony Reeves

The US government and the folks that feed off of it frequently say that the Internet is a battlefield (I thought that was love). They tell us that we have entered into a cyberwar. We’re all cyberwarriors (cybersoldiers?) and we need more cyberweapons to defend our cyberbunkers against cyberterrorists who threaten Truth, Justice, and the Cyber Way.

And I don’t doubt for a moment that state-sponsored attacks and exploitations happen as a logical extension of the electronic warfare that played such a large role in the 20th Century. Do the Chinese government and organizations associated with it try to get into, say, the Pentagon and White House and defense contractors? Of course they do. (And if you think that the converse doesn’t hold true as well, you should rethink that position.)

The military and government don’t want me working for them, and I share that sentiment. I don’t want to get into Full Metal Polo Shirt. But so many of us seem to romanticize the idea. Clearances abound to share threat data on malware that doesn’t relate to military but rather (at best) law enforcement related to organized crime. If you don’t have a fedora and a trench coat, some vendors won’t even sell to you. And all the vendors want to trumpet whatever connections they might have to DC, assuming that gives them lots of cachet.

I have tremendous respect for Richard Bejtlich, who lives this stuff every day and does a fine job explaining his well-thought-out positions. But he works for a large multinational that has assets and business interests with specific military or other national security implications. In other sectors, our ‘adversaries’ tend to the criminal side. (And I only wish that the folks on the other side of the data I see worked from idealism.) The simple fact of international involvement doesn’t automatically imply warfare; much of what we defend against comes from US allies or at least business partners.

Don’t fall into the trap of thinking that everyone must become militarized. When the computers rebel (“The bourgeois human is a virus on the hard drive of the working robot”), then we have something to fear.

→ Leave a comment

Posted in Manifesto

Tagged , , , , , , ,

I can haz MIRcon?

Posted on 2010-10-18 | 1 Comment

TLDR: Ninjas and beer. And security lulz.

Nothing livens up a beautiful fall day more than a bunch of geeks and suits sitting in a DC hotel meeting room talking about finding evil. Who doesn’t live for that sort of thing, amirite? So Mandiant decided that they’d get a bunch of incident response types in a room for two days, throw out some coffee and notecards and a projector screen, and see what happens.

'XIX Party Conference' by fotofreq

After having recovered from the beer tasting, intense questioning of product managers, and the ALDS (let’s go Rangers!), I can tell you that we haz moar ideaz than Taylor Swift has fashionable shoes. And that’s a lot, so some of this stuff will get a Killswitch Engage-style breakdown in future blog posts.

Kevin Mandia looks kind of like Nathan Fillion in a suit. He has nice hair, a rugged jawline, and a never-ending idiom factory lodged somewhere between those two things. (He should definitely get Captain Mal to play him in “Mandia: The Man, The Myth, The Legend.) His keynote speech put all of these on display, and we learned that a greenish-yellow Global Threatcon Severity Indicator means we’re all gonna die, man, game over.

Mandia-Fillion

Separated at birth?

Once we moved on to the panel discussion, things started to liven up. Well, that’s not true. The panel tried to stay away from too much management-speak (yeah, you know what I mean). For example, they talked about visibility and authority for incident response: how do you know if you’re compromised? What happens when you realize you are? You probably won’t reach a 100% solution, so shoot for achievable goals. I mean, I’m really happy for your incident response program, and I’m gonna let you finish, but GE has one of the best CIRTs of all time. OF ALL TIME!

Ahem. So your best threat intel might not come from your sekrit friends in the Illuminati^W military-industrial complex^W^W^W defense industry or critical infrastructure ISAC. Your best intel might come from the cases you’re already working. Pull on the threads you already have, see where they go, and soon your sweater will be undone.

'Loose Threads' by Chris Luckhardt

Then the sparks started to fly. Let me tell you, I thought GE and BAE were in a cage match (“two corporate security directors enter, one corporate security director leaves”). Richard “GE” Bejtlich got all Courage Wolf on us: neutralize the threat, take out the bad guys, RAWRAWRAWR! Then Ron “BAE” Davis was all about “hmmm, maybe that’s a bad idea, you could get the wrong person or go to jail”. And we were all, “ooooh, he TOLD you!” in the audience, and GE was all “ORLY”, and BAE was all “YA RLY”, but then they all smiled and reminded each other that we’re the good guys after all. And we all had ice cream with cherries on top. It was nice.

Michael Graven from Mandiant came up next on behalf of an “anonymous” customer to talk about tool integration. I’m going to call them “Anony Moose” from here on out. See, Mandiant is really proud of this technological terror they’ve constructed called Intelligent Response. It has a RESTful API that spits out well-defined XML and lots of other cool-sounding 2.0 kind of stuff. Anony Moose (wonder if they were in the room at all? nah, too obvious… OR IS IT) has a SEIM that automagically generates trouble tickets when it detects, er, trouble. In paradise. Or something.

So these tickets live in Request Tracker, which has a RESTful API. See some potential there? I guess Anony Moose did, too. They took off for every zig and matched the two of them up. I hear several steps and boxes with holes were involved. And when a ticket comes in, the system reaches out and touches someone^W the target system to gather lots of volatile data like ports, processes, recent audit logs, and all the sorts of things that a super-smart sekrit security agent will want to know. Good stuff and I bet it saves them time so they can kick back in their Fortress of Solitude, laughing at the evildoers stymied by their most excellent planning.

The next talk had ninjas. No, not real ninjas, because we couldn’t have seen them. And since ninjas just flip out and kill people, I’d be dead already. But these were malware reverse engineering ninjas, which is pretty sweet even if it isn’t Real Ultimate Power. They talked about generating Indicators of Compromise, which is how you can talk about targeted malware without actually giving it up to the antivirus vendors and getting useless MD5 hashes and registry key indicators. They do some awesome behavioral analysis in addition to standard sandboxing and static analysis. They can even look at DLL-based malware, and rocking that ain’t easy. When they get super-smart sekrit security agents who find this stuff in the field, then they tear apart the malware until they find the plans. I mean, uh, how it works and how to identify all variants.

'The Urban Ninja' by Tyson Cecka

After we had coffee to clear our heads from the sheer unmitigated awesomeness that was hex dumped assembly code in a screen magnifier, a panel discussion on information sharing ensued. Now, see, everybody wants somebody to love, and if they can’t get that, they at least want somebody to share their interest in a secure operating environment for truth, justice, and the Internet way. This leads to trust issues, because the first rule of the advanced persistent threat is that YOU DO NOT TALK ABOUT the advanced persistent threat. Though I suppose that’s only unless you have clearance, in which case you totally DO talk about the advanced persistent threat.

On that note, Sandia National Laboratories scares me. They have some project where FBI agents arrest the red team and interrogate people with Borg headsets that can totally read your mind. One minute, you’re doing an incident response cyber exercise, and the next minute a man in a black suit is all, “HALLO CAN I VIOLENCE YOUR BRAIN”.

Other organizations were a little more chill. They got back on the question of trusting other folks and maybe punching them in the nose if they talk about the advanced persistent threat (see?!), but then they realized they were harshing everybody’s mellow and just said we shouldn’t share the stuff that can hurt us. And threat data never expires. At least not if you’re a defense contractor.

Ever been playing Minesweeper and, just when you think you’re about to clear it in Expert mode, get that Blue Screen of Death? It’s not all bad. We learned about how crash dumps preserve your integrity. (Apparently that doesn’t work very well on Congress’s computer systems. HEY-YO!) Normally, these dumps of the process memory space and system debug data get sent to Microsoft, but if you don’t like Steve Ballmer getting his grubby paws on your data, you can instead direct all that stuff to an internal share. And hey, maybe you can get some good forensic data out of all those dumps. If you can, then you should probably grab all the dumps and drop them in a debugger. Highly situational, but it’s better than a mudkip.

Then we had beer. It was good.

Next morning, we had a sound check with Heavy D. Not really, but Michael Graven thinks he sounds like him even though he kinda looks more like River Cuomo back when Weezer made the Blue album. (Nothing but love for ya, Mike.)

MIRcon then hosted a real, actual, Man in Black. I don’t remember much except him putting on some sunglasses and asking us to look into his space pen LED. Oh, and that every investigation now relates to cyber (stop snickering, WoW players, you know that’s not what he meant). They want to get “IT” off the network, where I is “intellectual property” and T is your “treasure chest”. The FBI, or at least Assistant Director Gordon Snow, wants a culture shift so we quit putting the good stuff online where the Ebul Doers can get to it. Since the threat isn’t going away anytime soon and vulnerabilities get all the attention already, the best way to reduce your risk is to lower the value of the asset. (I assume he drives a Ford Focus, since his logic also says that the best way to keep your car from getting stolen is to drive a clunker.)

'How did that get here!?' by Purple Wyrm

Finally, Halvar Flake totally blew our minds. Honestly, he had me at “Hello, my name isn’t really Halvar Flake”, but then he set the hook when he went straight into “approximate maximum subgraph homomorphisms”. That is, imagine you have two street maps of different scale and resolution, but they actually represent more or less the same area. You’ll want to stretch and zoom and align and twist the maps until you can see where they overlap, so you can look at your wife and be all, “I told you we weren’t lost.” And that you found new families of malware.

But then, once you cluster these maps together, you can give out different signatures that tell everybody how to match up new maps. Nobody gets the same signatures, though, because (just like eating a Reese’s Peanut Butter Cup), there’s no WRONG way to perform an approximate maximum subgraph homomorphism. See, if it’s wrong, then it wasn’t really a homomorphism after all.

And by splitting things up, you don’t actually have to talk about the advanced persistent threat. That would lead to nose punchings, and anybody who would try to punch Kevin Mandia in the nose will definitely get a “NO WAI”. And probably their OWN nose punched. Because he’s Captain Hammer.

I did sort of lie. That wasn’t the final talk. But that was the last one I really noted, because after that we had a demo of MIR. And then Mandia and Bejtlich showed us their sweet incident response kung fu auditions. “I NEVER LOSE! YOU NEVER WIN!” Then real actual computer scientists handed out the Malies (think “mallies”, not “mailies” because the latter sounds kind of sexist and could create a hostile work environment). That was a one-shot deal, but many memes died to bring us that information.

And some lawyers came on to tell us about the legal costs of data breaches, but I had a plane to catch home.

For the future, I’d just suggest averaging out start times. 10:30am one day, 7:30am the next. Let’s split the difference and keep it at 9am, because ninjas and beer and baseball (let’s go Rangers!) and early mornings don’t mix.

If you want a more serious and professional look at MIRcon, go read Greg Pendergast’s summary over at the SANS Forensics Blog. And buy him a beer, because he’s a mensch.

→ 1 Comment

Posted in Conferences

Tagged , , , , , , , , , , , , , , , , , , , ,