PIX Parsing (Usable Logs!)
Posted by bmestep on October 22, 2007
If you have a Cisco PIX you are responsible for, don’t have a logging solution for it, or haven’t seen PLA, keep reading you’re going to be impressed. PDM is out and ASDM is an improvement but who wants to run https-server on a firewall?
There are plenty of commercial solutions available, some just perform log analysis and others attempt to perform event correlation. If you just want to solve the logging problem and are unable to implement a log collection system, take a look at PIX Logging Architecture.
I recently implemented Pix Logging Architecture (PLA) to help manage FWSM, PIX, and ASA logs. I hate having to cull through syslog files looking for traffic stats, tcp port usage, or accepted vs denied traffic information. This point and click world we live in has me spoiled, I don’t want to CLI everything - especially when I’m headed into a meeting to discuss who has access to the Oracle Applications database server.
What if you could get a daily snapshot of your firewall(s)? How about running queries instead of text-based finds? The following log messages are already supportedand reg-ex can be used to further extend the capabilities. Defining traffic types and being able to make graphs? Once you involve a database, you have to decide how often to purge the data.
If you like that and aren’t afraid of using open source, you’ll need apache, MySQL, perl (with some additional modules), syslog daemon or the syslog file, and one more Cisco PIX, ASA, or FWSMs. The OS is up to you, but Linux is highly recommended.
The PIX Logging Architecture [PLA] is a free and open-source project allowing for correlation of Cisco PIX, Cisco FWSM and Cisco ASA Firewall Traffic, IDS and Informational Logs.
PIX Log message parsing is performed through the use of the PLA parsing module or PLA Msyslogd module. Centralization of the logs is provided using a MySQL database, supported by a Web-based frontend for Log Viewing, Searching, and Event Management. PIX Logging Architecture is completely coded in the Perl programming language, and uses various Perl modules including Perl::DBI and Perl::CGI.
PLA’s website has been down since late December but Kris reports it’s BACK ONLINE! If you are interested in downloading the software or documentation, you can find it on SourceForge but your best bet is to go directly to the source itself: PIX Logging Architecture.
I’m working on uploading the documentation and some tweaks to pull in Cisco IPS Event Viewer data into the IDS tab onto my other website, but haven’t been able to reach the original author (Kris) yet.
How does PLA work?
The PLA software is divided into a parsing piece, database server, and web front-end. Depending on your security policy, network zoning, and server capabilities, you can run this on one server or spread it across several. I was fortunate enough to have access to a dual processor duo2 server, so I implemented everything on the same Linux server.
The underlying architecture is very clever. Message formats are loaded into memory and the syslog file is tailed (via perl module). Messages are flagged by type and processed accordingly, in near-real-time:
The PIX Logging Architecture parsing module, which is responsible for extracting the necessary fields from the PIX system log messages, has been extended to gather new information including, but not limited to, Translations (Xlate’s), Informative Log Messages (i.e. PIX Failover, PIX VPN Establishment, PIX Interface Up/Down, PIX PPPoE VPDN establishment and the like). All the parsing information needed by the PLA Parsing Daemon (pla_parsed) in order to extract data from the logs is now stored in the database, allowing for easy updates of the supported log messages without having to replace the parsing scripts. Moreover, the PLA Parsing Daemon runs as a daemonized Perl process in the background and reads straight and in quasi real-time from the system log files, so no more need to create crontab jobs like before and having to restart syslogd all the time.
The PLA Team has created detailed documentation for installing, tweaking, and supporting PLA. [edit: Google cache of documentation. PLA Documentation can be downloaded at SourceForge].
In order to get started, you’ll want to read through the first few sections to get a feel for what will be required and figure out what you’ll need in your environment. For example, if you run standard syslogd the parsing of the syslog file works one way and if you run ksyslog or syslog-ng you’ll need to make adjustments. Section 5.5 of the documentation covers tweaking the regex for parsing other syslog engine formats. The documentation explains centralized installation versus distributed, as well as what ports and services will be needed on the server(s).
Security is obviously a concern when processing security log files. Securing the MySQL database access and Apache access is recommended. I would also recommend securing the server itself, if you’re new to Linux this may help.
Enjoy!
October 23, 2007 at 6:06 am
[...] here for [...]
October 28, 2007 at 2:11 pm
[...] wrote an interesting post today on PIX Parsing (Usable Logs!)Here’s a quick [...]
January 14, 2008 at 7:46 am
Dear Sirs,
I have a tremendous problem with PLA.
I have all the MySQL, Perl and its moduls, pla is running, but it is not putting anything in the DB… I have written to PLA author, but so far no go…
Can you please point me to some forum, etc?
PS I really do not agree that the documentation is great. It is quite confusing in places…
January 14, 2008 at 10:58 am
PLA Documentation page contains a mailing-list you can subscribe to the support mailing list. Have you emailed: pixla-support-owner@logging-architecture.net
Looks like their website is having some issues - I’d be happy to help - tack @gmail.com onto the ID I am posting with, to email me directly and I’ll try to help you out.
-B
January 23, 2008 at 5:03 pm
Velda - I also ran into the problem of not getting log data into the database after setting everything up. As it turns out, I was using syslog-ng and not syslogd. Apparently syslog-ng cuts one of the date/time stamps that are required by the parsing script. I installed on another machine using syslogd and everything worked as advertised. I am very impressed with this tool.
January 23, 2008 at 5:55 pm
The online documentation had a section for tweaking the parser’s regex to use other syslog engines (see section 5.5). I’ve added a link to Google’s cached version of the online documentation. I’ve tried emailing Kris to see if hosting the documentation at my other site would be okay, since the main site is down.
January 30, 2008 at 9:34 am
Hey guys,
Im interested in implementing this. But really..im not finding the supported platforms list? Can anyone give me some details as to what OS that used and the versions to get this up and running? jrego@lynn.edu
February 25, 2008 at 4:35 pm
[...] We recommend PIX Logging Architecture (PLA) for organizations interested in seeing what’s going inside their Cisco PIX firewall as a strong open-source, low investment solution. (Excellent write-up on PIX Logging Architecture.) [...]
March 17, 2008 at 8:47 am
I like this tool. It’s simple and powerfull.
But there is a little problem with “pla” and his history management. In my implementation, the daemons receives logs from four ASA firewalls. In three weeks traffic_log table has reached 20 Million records. It’s clear that the amount of log messages is not the problem, but the data model doesn’t take care about performance when making SQL querys on this case.
I have changed the type for this table (to innodb), including table partitioning (Mysql 5.1) and tweaking some performances parameters, but can’t make it work faster than yesterday.
It will be great if anybody can share similar experiences.
Thanks.
March 17, 2008 at 1:47 pm
Patrux, if you want to share what you’ve already done for changing over the db structure and SQL, I would be glad to take a look at the resulting changes needed in PLA.
I agree larger record counts will be slower; however, the SELECT statements being run by PLA are single search transactions with nested conditions and unless you’re using this in a multi-user environment it would seem these SELECT statements would fare better in the MyISAM format than InnoDB.
One could implement some degree of normal-form on the database on IP’s and Port’s but I’m not sure how much that’s going to buy you.
March 18, 2008 at 5:01 am
Hi all,
For information, after 3 months of being offline, the PLA website is back online: http://www.logging-architecture.net.
Patrux and Bmestep, seen as I am far from an expert on MySQL, I would like to look more closely at the MySQL recommendations to speed up the queries and matching you have done so I can include them in the next version of PLA.
Thanks,
Kris
March 18, 2008 at 2:47 pm
hey!
that’s cool.
I’ve spent some time trying to speedup PLA without good results.
I think that the problem (in my case) is that we’re running “syslog server” and the database on the same system.
About the database changes, i’ve made changes to storage type on MySQL, adding partitioning and fixes (i hope) to some querys.
I changed expression:
log_time>’expr’ and log_time<’expr’
for
log_time between ‘time1′ and ‘time2′
while i’m using partitions by “range” (to_days) this sentences uses only partitions that contains records in this interval of time. The original sentences uses all partitions.
Now i’m working to move the database and the front-end to another system.
I hope that we’re having news about PLA!.
March 18, 2008 at 7:44 pm
I have a traffic_log database with over 148Million records, I can run some tests on the queries to see which ones perform faster, in the case of the one you mentioned above.
Look for a new post to capture ideas on enhancing speed / performance of PLA.
March 19, 2008 at 12:59 am
Patrux, Bmestep,
Thanks for the updates. If you want you can contact me via email on kris [at] logging [dash] architecture [dot] net. A traffic_log db of 148 Million records is not bad, how’s that fairing up for you so far? Are you using many display / parse filters? With regards to the speed of the MySQL queries, because of the display filtering being enabled and perfoming a JOIN, it makes the query a little slower. I’ll have to look around to see whether there’s a more effective way of handling these queries.
Bmestep, I like your write up on using rsyslogd with PLA, do you mind if I put a link to your write up on the PLA website?
Kris