Everything about logs is big. Big in terms of:
- Endpoints and devices to collect from
- Events per second and the bandwidth, CPU and general throughput needed to process them
- Storage required to archive logs
- Quantity of different log formats
- Complexity involved in correlating multiple events to find the needle in the haystack
- Number of different use cases and interested parties
In recent conversations, an Enterprise security team leader said “we don’t want a lot – we want it all” with regard to which logs they needed to be able to collect and analyze. I thought that was a great way to put it. If you want to find threats, be able to thoroughly investigate incidents and fully determine the extent of a breach, if you want to provide forensics for any foreseeable situation and if you want to ace compliance reviews, you need to collect every log with security value.
In response to that statement, I sometimes hear folks push back with comments like “We are already monitoring and getting more alerts than we can follow up on” and “our SIEM can’t handle anymore load; we can’t afford to upgrade our SIEM or it won’t scale to that level”.
All of those may be true, but none of them need to be an issue. Just because you collect a log doesn’t mean you have to send it through your SIEM. It doesn’t even mean you have to monitor or review it at all. At a minimum, archive the logs so that if you ever do need them, they’ll be there. You can’t go back in time to get a log that got overwritten. Beyond barebones archiving, the next step is to have basic search capability – even if it’s with an open source big data search product. You’d be amazed how much simple search capability across every log on your network will do for you when you are running an investigation or tracking down an issue.
So, am I suggesting 2 different ways of handling log data? At least 2. Most organizations of size cannot send every log through SIEM. So, send the most important logs and, better yet, the highest value events, through your SIEM and UBA technologies. But don’t neglect the rest of your logs. Get all of your logs into some kind of affordable archival and search repository.
There are many different options for how to establish your overall log flow.
Or some organizations choose to forward logs from SIEM to their long-term archival solution like this
In this real training for free session, we will explore and contrast these and other options. There are many ways to skin the cat and it’s important to separate event log management – at least conceptually – into its constituent parts: collection, monitoring, analytics, archival, etc.
Log collection is particularly important to consider physically separating from your analytics and archival technologies because these can come and go but your logging pipeline should remain.
Quest Software is sponsoring this webinar and Bryan Patton will briefly show you how InTrust is an affordable but powerful place to send logs and how Change Auditor can feed high value security events to your SIEM without all the noise of the native logs.
Please join us for this real training for free event.