The next generation of Security Incident Event Management Systems (SIEMs) are taking full advantage of big data systems (i.e. hadoop), which for a small or medium company, it’s a lot of overhead to build and maintain, but for large companies, it’s the perfect size, and upgrading is seamless.

The three jobs of a next generation SIEM as follows. Any solution that doesn’t do all three is not a complete solution, and you would need more than one solution to cover the missing parts:

  1. Real-time basic correlation. Looking for single logs, or combinations of logs that correlate into a defined security problem. Each log that comes in is checked against a set of rules, and matches are alerted on. This can be general things like impossible travel (two authentications from the same user, from two different locations, without enough time to have legitimately traveled between them). Or highly customized things like VPN login from a country that you don’t do business in, or interactive login from a scripting/functional account. In medium/large companies, you would have SIEM authors who would be writing these correlations.
  2. Real-time Analytics. The scope is individual logs, as they come in. The logs are evaluated against analytical models. This varies widely by the model, but could involve anomaly detection, outlier detection, or one of many other types of model, and may involve unsupervised machine learning. An example of this could be User Behavioral Analytics, where, if user Mike normally works 9-5 M-F from Canmore, all of his access is clustered tightly around those values. If he logs in at 8:50, that’s fine, it’s not too much of an outlier, but if he logs in on Saturday at 2am from Detroit, that would be a significant outlier. In large companies, you would have Data Scientists building and maintaining these models.
  3. Bulk Analytics. This is the same as Real-Time Analytics, but is processed in bulk instead of real-time, against a potentially large amount of data. This is commonly used for training models, for finding patterns that are difficult to find in real-time, or taking currently known threat information, and applying it historically to see if anything was missed. An example of this is looking for Command and Control (C2) beaconing. That is regular connection patterns between (possibly infected) hosts and C2 servers. Doing this in real-time would be very difficult because of the amount of information that you would need to keep in memory, but in bulk is much easier, since you can look at all of the relevant connections at the same time. These are typically done as map-reduce jobs on the cluster.