Blog Archives

Imperva Placeholders

I had an email asking what placeholders I usefor logging platform integration. Rather than reply in a comment or email, I thought I’d just make a post out of the response.

Looking at placeholders, here are some of the ones I use the most:

  • ${Alert.dn}  this is the alert id
  • ${Alert.createTime} this is the time the ALERT was created (note this can be misleading)
  • ${Alert.description} this is bound to the alert, so you may see “Distributed” or “Multiple” appended due to aggregation of events
  • ${Event.dn} this is the event (violation) id
  • ${Event.createTime} this is the time the EVENT was created (this is when the event happened}
  • ${Event.struct.user.user} this is the username from a web or database action
  • ${Event.sourceInfo.sourceIP}
  • ${Event.sourceInfo.sourcePort}
  • ${Event.sourceInfo.ipProtocol}
  • ${Event.destInfo.serverIP}
  • ${Event.destInfo.serverPort}
  • ${Event.struct.networkDirection} which way is the traffic flowing that triggered the event?
  • ${Rule.parent.displayName} this is the name of the Policy that was triggered

There are other placeholders you can leverage, but these are the core I start with. I like these because they’re used on the web gateway AND the database gateway. This lets me have a consistent intelligence feed to my log monitoring platform and my SIEM product.

The trick here is that I can see how may events roll up underneath a single Alert. In the syslog feed, I can track the duration of an attack as well as tell you when I last saw the activity, because I track Alert.createTime and Event.createTime.

There are lots of options for how you build your syslog feed:

  • You may be interested in the response time of the query or web page
  • Perhaps the response size is of concern to you
  • You may treat threats differently depending on where they occur in a database table or URL
  • You may be interested in the SOAP action or request

Last but not least, in addition to security events you can also push system level events in the same manner using different placeholders.

  • Configuration events can be syslog’d on complete with the user making the change
  • Gateway disconnect messages can be sent via syslog (snmp might be better, but you need to load the custom OIDs)
  • Excessive CPU or traffic levels can be sent via syslog

How are you using placeholders?

Imperva: Alerts and Events

I received some emails overnight on the Imperva DIY Syslog posting asking when to use the alert placeholders versus the event placeholders.

For anyone not familiar with the Imperva SecureSphere platform, the system has a handy feature that provides aggregation of events on the SecureSphere management server detected by the gateways. This works whether you’re using the web or database gateways but for today I want to focus on the relationship between the data coming from the gateways and the aggregated data on the manager,  I’ll let ImperViews get into the other details – you can read more in the Imperva documentation.

The first thing you have to take note of is the Imperva hierarchy for violations/events and alerts. When the Imperva detects a condition that meets the criteria of a policy, whether that’s correlation, signature, profile, custom, etc., a violation is triggered on the gateway and fed to the management server. Everything in the management server for reporting and monitoring builds off this violation/event detail from the gateway, the gateway is where the enforcement and detection takes place so that should make sense. This is how we know the gateway is taking action on our behalf!

Assuming you haven’t disabled aggregation on the SecureSphere settings, each violation is aggregated into an alert. There are several criteria that the management server uses when aggregating a violation, so you’ll want to check the documentation for your version. The basic idea is that the SecureSphere manager will aggregate similar violations against a server group, an IP Address, a URL, a policy, or some combination of thereof in a 12 hour window. An alert in SecureSphere will have at least one violation/event tied to it, but depending on your aggregation settings it may have more.

So???

So! When you push security events to an external log monitor, you have to decide if you just want the initial Alert information or if you want each violation that occurs! If you build the Action Interface using ALERT Placeholders you’ll only get the Alert data with no additional details in the underlying violation/event stream. This could be problematic, if you’re trying to figure out if something is still going on because remember the SecureSphere aggregates violations under a single Alert for up to 12 hours!

In addition to using the correct placeholders, you also have to enable the “Run on every event” checkbox in the Action Interface/Action Set.

I tend to mix the Alert and Event placeholders so that I get relevant Event details wrapped in the Alert context. I see no reason to make my logging solution work extra hard to establish the same correlation of the Events into Alerts that SecureSphere does automatically.

How do you manage your SecureSphere alerts and events?

Imperva’s DIY syslog format

I have had the fortune to support a few WAF installations, my preference is Imperva’s WAF solution. For any security product, being able to know what it’s doing and what is going on within the product is as important as the actual security being provided.

One of the features of Imperva’s solution that I find tremendously useful in an enterprise setting, and possibly an MSSP as well,  is the ability to construct custom syslog formats for triggered alerts and system events in almost any format. I like to think of this as a Do-It-Yourself syslog formatter because the feed can be built and sent anywhere, using any number of options. More importantly, the feed can be bundled with specific policies or event types to provide limitless notification possibilities that often require professional services engagements to develop and implement.

In Imperva terminology, any policy or event can be configured to trigger an “Action Set” containing specific format options for among other things syslog messaging. If your logging platform (PLA) or SIEM requires a specific format, there’s a very strong chance that, with no more effort than building a policy, you can build the ${AlertXXX} or ${EventXXX} constructs necessary for your needs.

You can model the alerts to look like the Cisco PIX format, ARCSight’s CEF format can be used, or you can make your own as I’ve done in this screenshot:

Basic Syslog Alert Format

Basic Syslog Alert Format

In addition to allowing customized messaging format, Imperva’s SecureSphere platform allows unique message formats and destinations to be specified at the policy and event level. For example, a “Gateway Disconnect” or ” throughput of gateway IMPERVA-01 is 995 Mbps” message can be sent to the NOC’s syslog server for response, while XSS or SQL Injection policies can be directed to a SOC or MSSP for evaluation. Additionally, the “Action Set” policies can be setup so that the SOC is notified on both of the  messages above as well as security events.

The configuration of the custom logging format is very straightforward, using placeholders to build the desired message format.  The document “Imperva Integration with ARCSight using Common Event Framework” provides a number of examples, including a walk-through for building a syslog alert for system events, standard firewall violations, as well as custom violations. The guide is directed at the integration with ARCSight.

Depending on the version of Imperva SecureSphereyou are running / evaluating, the alert aggregation behavior will differ. Newer versions (6.0.6+) better support SIEM platforms with updated alert details, where older versions push syslog events on the initial event only.

You can request a copy of Imperva Integration with ARCSight using Common Event Framework to get additional ideas on customizing your syslog feeds for your SIEM product.

Getting more from your WAF (Sensitive URL Tracking)

I have had the fortune to support a few Imperva installations, alongside other WAF solutions. I would like to illustrate one use for logs available on the Impervaplatform that can be leveraged to augment website trend reports and monitor “exposure” on key URL’s.

If you’re not familiar with the Imperva platform, it is possible (as with other WAF vendor’s products) to build custom policies that must match specific criteria and upon triggering these events can feed data into various syslog feeds. The entire purpose of a WAF is to protect your web application from threats, although some argue this point, so it stands to reason there may be facets of a given web application that are more sensitive than others.

Take for example the check-out page for an online retailer where the customer enters credit card data and confirms their billing information. This location of a web application might benefit from heightened logging under certain conditions by a Web Application Firewall, such as: forced browsing, parameter tampering, XSS, Server Errors, etc. The application may be vulnerable to fraud activities, the business may want to keep a tab on who’s accessing these URLs, or there some other risk criteria than can be measured using this approach.

Traditional webserver logs will provide: client information such as user agent info, username, source ip, method, access URL, response time, response size, and response code. The logged data sits in the access log file on the specific web server by default, but this information is for the entire website.

The Imperva SecureSphere can provide some of the same information: username, IP, Port, user-agent info, accessed URL, response size, response time, etc – but in addition, the Imperva can track whether the session was authenticated, correlated database query (if you have Imperva database protection deployed), SOAP information, security details relevant to the specific policy. The kicker is that this can be sent in a format configured by the admin to a syslog listener in a format supported by web trend tools or SIEM products without engaging professional services.

I’m not advocating the replacement of web server logs for trend analysis, but I am suggesting the deployment of targeted logging for sensitive areas inside an application where this information would prove useful either in a fraud capacity, security monitoring capacity, or even in an end-to-end troubleshooting capacity where a WAF would have visibility beyond traditional network tools from the frontend of a N-tier web application. Deviations in response times, excessive response sizes, and unauthenticated access attempts to sensitive URLs are ideas that come to mind for leveraging the visibility a WAF can bring to the table.

PIX Logging Architecture is Back Online

Great news, after a brief recess, PIX Logging Architecture is back on the NET!

Be sure to checkout the screenshots for features / selling points and import the latest syslog-message database if you’ve not done that since installation. Remember PLA handles the following Cisco Security Devices:

PIX Logging Architecture v2.00 supports log messages from the following devices:

  • Cisco ASA (TESTED AND CONFIRMED)
  • Cisco PIX v6.x (TESTED AND CONFIRMED)
  • Cisco PIX v7.x (TESTED AND CONFIRMED)
  • Cisco FWSM 2.x (TESTED AND CONFIRMED)
  • Cisco FWSM 3.x (TESTED AND CONFIRMED)

PLA Documentation

PLA Screenshots and more PLA Screenshots

PLA: Latest PLA syslog message support

Support for alternate forms of syslog daemons can also be found here for parsing rsyslog.

Welcome back Kris!

PIX Logging Architecture

Tweaking PLA: Using rsyslog

PLA (PIX Logging Architecture) uses regular expressions (regex) to parse syslog messages received from Cisco firewallsand comes pre-configured to process standard “syslogd” message format. Most current Linux distributions ship with rsyslog (able to log directly to MySQL) while some administrators prefer syslog-ng.

The installation documentation distributed with PLA assumes a familiarity regex, so here you’ll see how to tweak PLA to parse your rsyslogd log file.

Perl is used to parse through the syslog message looking for matches to message formats described in the syslog_messages table in the pix database. The processing script pla_parsedcontains a regex pattern that must be matched in order for the processing to occur. The applicable section is:

### PIX Firewall Logs
### DEFINE YOUR SYSLOG LAYOUT HERE!
###
$regex_log_begin = “(.*):(.*) (.*) (.*) (.*) (.*) (.*)“;
$var_pixhost=3;
$var_pixmonth=4;
$var_pixdate=5;
$var_pixyear=6;
$var_pixtime=7;

Here, the variable regex_log_beginneeds to match up all the log information up to the PIX, ASA, or FWSM message code in order to understand date, time, and host for these messages. Take a look at the provided sample log entry, everything in red needs to be picked up by regex_log_begin while the remainder is standard for Cisco firewalls:

Oct 21 23:59:23 fwext-dmz-01 Oct 21 2006 23:58:23: %PIX-6-305011: Built dynamic TCP translation from inside:1.1.1.1/2244 to outside:2.2.2.2/3387

Explaining the operation of regex and wildcards is beyond the scope of this article; however, numerous guides have been written to fill the void. In our case, adjusting the default regex to match rsyslog is straight forward after noting which characters match which pattern, again we’re working with the basics of regex here – nothing fancy.

Take this sample rsyslog entry and notice the difference from the standard syslogd format:

Feb 21 10:59:32 Feb 21 2008 10:59:32 pix4us : %PIX-6-110001: No route to 1.1.1.1 from 3.4.5.6

Feb 21 10:59:32 Feb 21 2008 10:59:32 pix4us
Oct 21 23:59:23 fwext-dmz-01 Oct 21 2006 23:58:23

Here, the rsyslog entry includes the date twice and then the hostname of the log source versus the default format expected by pla_parsedof date hostname date. The original regex is set to pickup the first time entry’s “minutes and seconds” and picks up the next 5 words/entries separated by spaces:

$regex_log_begin = “(.*):(.*) (.*) (.*) (.*) (.*) (.*)“;

 Oct 21 23:59:23 fwext-dmz-01 Oct 21 2006 23:58:23

In order to process rsyslog, this will have to be changed. The initial (.*):(.*) is used to set a starting point in syslog message string. Since this new rsyslog format includes two date entries before the host name, the following can be used to allow pla_parsedto “see” the new syslog message string:

$regex_log_begin = “(.*):(.*) (.*) (.*) (.*) ((.*):(.*):(.*)) (.*)“;

Feb 21 10:59:32 Feb 21 2008 10:59:32 pix4us

The regex starts out the same, but looking at the colors you will notice the location of the information needed by pla_parsed to determine date, time, and host has moved. This time we used “(.*):(.*)” and “((.*):(.*):(.*))” to force a match on the time elements.

As a result of this change, the variables listed below the regex pattern must be modified to tell pla_parsed which (.*) contains which element:

$regex_log_begin = “(.*):(.*) (.*) (.*) (.*) ((.*):(.*):(.*)) (.*)“;

$var_pixhost=7;
$var_pixmonth=3;
$var_pixdate=4;
$var_pixyear=5;
$var_pixtime=6;

The numbering happens left to right and the color coding should help this make sense. The ()’s around the grey time entry are grouped together and count as one match/entity, the sixth variable. This same approach of keying off the timestamping can be applied to pla_parsedin order to allow processing of syslog-ng, ksyslogd, or any other syslog message format.

Need help with a different format? Have problems getting your PIX logs loaded? Paste in a sample message from your syslog server (IP Addresses santized please) in a comment below.

PIX Parsing (Usable Logs!)

If you have a Cisco PIX you are responsible for, don’t have a logging solution for it, or haven’t seen PLA, keep reading you’re going to be impressed. PDM is out and ASDM is an improvement but who wants to run https-server on a firewall?

There are plenty of commercial solutions available, some just perform log analysis and others attempt to perform event correlation. If you just want to solve the logging problem and are unable to implement a log collection system, take a look at PIX Logging Architecture.

I recently implemented Pix Logging Architecture (PLA) to help manage FWSM, PIX, and ASA logs. I hate having to cull through syslog files looking for traffic stats, tcp port usage, or accepted vs denied traffic information. This point and click world we live in has me spoiled, I don’t want to CLI everything – especially when I’m headed into a meeting to discuss who has access to the Oracle Applications database server.

What if you could get a daily snapshot of your firewall(s)? How about running queries instead of text-based finds? The following log messages are already supportedand reg-ex can be used to further extend the capabilities. Defining traffic types and being able to make graphs? Once you involve a database, you have to decide how often to purge the data.

If you like that and aren’t afraid of using open source, you’ll need apache, MySQL, perl (with some additional modules), syslog daemon or the syslog file, and one more Cisco PIX, ASA, or FWSMs. The OS is up to you, but Linux is highly recommended.

What is PLA?

The PIX Logging Architecture [PLA] is a free and open-source project allowing for correlation of Cisco PIX, Cisco FWSM and Cisco ASA Firewall Traffic, IDS and Informational Logs.

PIX Log message parsing is performed through the use of the PLA parsing module or PLA Msyslogd module. Centralization of the logs is provided using a MySQL database, supported by a Web-based frontend for Log Viewing, Searching, and Event Management. PIX Logging Architecture is completely coded in the Perl programming language, and uses various Perl modules including Perl::DBI and Perl::CGI.

Where is PLA?

PLA’s website has been down since late December but Kris reports it’s BACK ONLINE! If you are interested in downloading the software or documentation, you can find it on SourceForge but your best bet is to go directly to the source itself: PIX Logging Architecture.

I’m working on uploading the documentation and some tweaks to pull in Cisco IPS Event Viewer data into the IDS tab onto my other website, but haven’t been able to reach the original author (Kris) yet.  

How does PLA work?

The PLA software is divided into a parsing piece, database server, and web front-end. Depending on your security policy, network zoning, and server capabilities, you can run this on one server or spread it across several. I was fortunate enough to have access to a dual processor duo2 server, so I implemented everything on the same Linux server.

The underlying architecture is very clever. Message formats are loaded into memory and the syslog file is tailed (via perl module). Messages are flagged by type and processed accordingly, in near-real-time:

The PIX Logging Architecture parsing module, which is responsible for extracting the necessary fields from the PIX system log messages, has been extended to gather new information including, but not limited to, Translations (Xlate’s), Informative Log Messages (i.e. PIX Failover, PIX VPN Establishment, PIX Interface Up/Down, PIX PPPoE VPDN establishment and the like). All the parsing information needed by the PLA Parsing Daemon (pla_parsed) in order to extract data from the logs is now stored in the database, allowing for easy updates of the supported log messages without having to replace the parsing scripts. Moreover, the PLA Parsing Daemon runs as a daemonized Perl process in the background and reads straight and in quasi real-time from the system log files, so no more need to create crontab jobs like before and having to restart syslogd all the time.

The PLA Team has created detailed documentation for installing, tweaking, and supporting PLA. [edit: Google cache of documentation. PLA Documentation can be downloaded at SourceForge].

 In order to get started, you’ll want to read through the first few sections to get a feel for what will be required and figure out what you’ll need in your environment. For example, if you run standard syslogd the parsing of the syslog file works one way and if you run ksyslog or syslog-ng you’ll need to make adjustments. Section 5.5 of the documentation covers tweaking the regex for parsing other syslog engine formats. The documentation explains centralized installation versus distributed, as well as what ports and services will be needed on the server(s).

Security is obviously a concern when processing security log files. Securing the MySQL database access and Apache access is recommended. I would also recommend securing the server itself, if you’re new to Linux this may help.

Enjoy!

Log Management 101

If you are in any way involved with technology, you have at one time or another dealt with logging. You either didn’t have what you needed, spent hours looking a needle in the haystack, or had to go find someone to help you get what you needed.

In networking and security circles, logging is usually a hot-button topic. I’ve spent many man-months working with logs, investigating logging solutions, praying to God for a solution to logging, tweaking open-source logging programs, and discussing logging technology with various vendors. I’ve collected some of the more relevant articles on the topic and have some additional commentary / insight.

First and foremost, the primary problem of Log Management is often not the log(ging) portion – most systems spew limitless log data – the problem is the management of the log(ged) data. You’ve heard 1000 times, management styles vary by manager. Management of network, servers, and security differ wildly. Networks and servers are managed by exception, they get attention when they need it otherwise they just purr and no one bothers them (until Patch Tuesday or until Cisco has a bad day). Managing security is altogether different, because security for a given organization is the net sum of all the parts of said organization, and in order to properly manage security, information from all the parts must be reviewed on a routine basis.

This routine review of information is where the problem comes in and it is why nearly every regulation, guideline, or contract stipulation (read PCI DSS) has mandatory language about reviewing logs. Again there is seldom anything about requiring systems to output logs, because that is rarely the problem. Technology is often times misapplied in attempts solve human-created problems, log management is an ideal instance for technology to come to our rescue. How else would a sane person cull through gigabytes or terabytes of log data from ten’s or hundred’s of systems or network elements? <Don’t answer that question, I’ve done it myself .>

Logging resources

Dealing with the actual logs and delving into whether you want to collect log data, consolidate the log data, correlate log data, and or perform event management usher in the next set of issues / challenges when dealing with logging. Originally logs might have been aggregated onto one server or they might have been aggregated by log severity using syslog-ng or syslogd. Network or systems management systems (CiscoWorks or HP OpenView) might have received logs and performed parsing or filtering on key events that drove alerts and alarms but these systems likely discarded the log data itself, so investigating the root cause of an alert or incident became next to impossible.

Compliance came to the scene and brought innovation in log collection and log management; software became available to manage logs from multiple sources but you had to pick your poison: Windows with MS SQL back-end, Solaris or HP-UX with Oracle, or Linux and MySQL. Aggregation and consolidation were solved by tiered deployments and agents were usually installed to siphon off Windows Event Data. Security hit the mainstream and event correlation was the big buzz word.

The problem was how the intensive resources necessary to store the log data competed with the intensive resources necessary to correlate dissimilar log formats to produce alerts or suppress logged events. Gartner and others coined the the term SIEM, which combined log collection and event correlation, and a new market arrived. Most SIEM (SIM, then SEM, finally SIEM) are really good at the collection piece (historical) or really good at the correlation (real-time) piece, while few are good at both. Go, go Magic Quad. rant! If you hadn’t already noticed, I’m not a fan of combining these technologies (I don’t mix my food while I’m eating either). I like my logging solution to have plenty of evidence preservation goodness and I don’t want it muddied because a correlator had to normalize the data before it could parse, alarm on, or display the log data.

Some of the options for solving log management challenges

Just scratching the surface, more please?