At my organization, much of the threat intelligence we receive consists of one-off reports that require some manual processing. But we also receive a weekly consolidated list, which primarily consists of IP addresses, domain names, URLs, file names, and MD5 hashes in a CSV file.
While we’ve considered bringing threat intel data into our ArcSight SIEM implementation for some time, until lately we did this manually as well. This can cause lots of difficulties: staff falling behind due to the volume of other work we perform, individual variations in process, and lack of ongoing correlation. So I decided to implement a small script to make our lives easier.
Originally, I’d planned on modifying ArcOSI for this purpose. ArcOSI scrapes several open source intel feeds, runs a regex on them for IP addresses and domain names, then streams the data in ArcSight’s Common Event Format (CEF) over syslog for proccessing and correlation. As it turned out, my data have enough consistency that I didn’t need all the logic and configuration that ArcOSI provides. I ended up gutting the whole thing and writing a Python script more or less from scratch, keeping only the four lines that actually communicate with the Collector via UDP. (The csv module in Python sped this up tremendously.)
I also found that CEF actually makes this fairly easy. Basically, you just send a syslog message as follows:
CEF:Version|Device Vendor|Device Product|Device Version|Signature
The Name field should be fairly generic (“Port scan”, not “Port Scan from 10.1.1.1 against 10.2.2.2″). The Extension field actually consists of key-value pairs. ArcSight includes some predefined keys like src & dst (source address / destination address, and you can define additional keys plus include a detailed message. So, in my script, I emit two signatures: “Known Malicious IP Address” and “Known Malicious Domain Name”. I include some other data in the Extension fields such as the indicator’s tracking ID and the date that it appeared in our intel feed.
Now we just need to create an Active List in ArcSight ESM for these indicators and correlate appropriately. When an event comes in that correlates to one of these indicators, we’ll generate a notification (at least as a first pass). We also have an engineer building out appropriate reporting, trending, and dashboards.
Think about your own intel feeds and whether it makes sense to use ArcOSI or write your own script to parse the feeds into CEF.