FLoP - Fast Logging Project for Snort

Actual version: 1.6.1

NEWS

Lastest version of FLoP
Patch for snort-2.9.0.5
FLoP - Mailing list
Changes in version 1.6.1
Changes in version 1.6.0
Changes in version 1.5.1
Changes in version 1.5.0
New patch for snort-2.4.3

Changes in version 1.4.1

Changes in 1.4.0 since version 1.3.0
Changes between 1.0 and 1.2.3:

Readme files
Survey of the project
Documentation
Download
Summary
License
Finally

NEWS

Lastest version of FLoP

Here is the latest version of FLoP: FLoP-1.6.1.tar.gz

Patch for snort-2.9.0.5

Here is the patch for the actual snort version: snort-2.9.0.5_patch

FLoP - Mailing list

There exists now a mailing list for FLoP: flop@lug-erding.de
To subscribe send an email to flop-request@lug-erding.de with the subject subscribe.

Changes in version 1.6.1

Now more than one unix socket can be used by snort.
MySQL reconnects were used if available
fpg has now much more features and even pcre support thanksto Jürgen Leising
servsock can now check periodically if the clients are still alive
a lot of bug fixes

Changes in version 1.6.0

Added option to use cache for sig_id values. This cached values are stored in a red-black balanced binary tree and have a high impact on the database INSERT rate. If the sig_id is in the cache then no SELECT is necessary. In order to use this option you have to activate it with the configure option --enable-cache.
With PostgreSQL you can now use the PREPARE statement to enhance the database performance. But all tests so far showed only a slightly enhancement if there is one at all. You have to use the configure option --enable-PREPARE.
servsock checks now the functionality of FullPayload and Reference in separate functions to avoid a roll back of the sensor entry in the case they are missing.
servsock writes every 1000 alerts a status message if they are read in from or written to the swap file.
All invalid data from a swap file get now ignored whereas the valid ones are stored in the database.
The maximum number of concurrent sockserv-servsock combinations is now adjustable via the configure option --with-maxclients=#sensors.
Statistics are now generated via the control thread, the SIGALARM is no longer used.
Added further checks to ensure how many clients are connected.
snort adds now a tag to the alert structure written to the unix domain socket. Additionally another tag is used to indicate an end or restart of the snort process. This was necessary to avoid an out of order condition if snort gets restarted between the transmission of the alert data and the payload.
The vendor map to identify the manufacturer of an ethernet device got 1500 additional entries.
The restart and termination routines in servsock were reworked. Note: On linux-2.4 with the old linuxthreads this may not work as expected.
Support for Base64 is added to getpacket
getpacket is now able to create linux cooked mode pcap files (thanks to Juergen Leising).
EventOffset was added to sockserv to avoid problems with the Reference if snort got restarted but sockserv not.
rules.pl is now able to access the database via TCP/IP instead of the unix socket. (Some people like to run the database in a chroot() environment...)
rules.pl got extended to support scheme 107.
Some memory leaks in fpg got removed.
A bug with negative distance values was removed (thanks to Juergen Leising).
db-cgi.pl got extension to support base64 (again thanks to Juergen Leising).
Documentation got reworked and extended.
A lot of minor bug fixes.

Changes in version 1.5.1

A problem with the sensorname is corrected, it could happen that parts of the last sensor appeared after a newline.
sockserv can now add the interface name (option -I).
Some adjustments to thread code to avoid problems with OpenBSD-3.8. (It seems they have fixed some problems with threads in earlier versions so not all work arounds are necessary.)
The termination of servsock was rewritten, on rare situations it could happen that some processes hung or writing of the buffered alerts to the swap file was interrupted.
A time delay was added to sockserv if the connection to servsock was interrupted.
Empty alerts are now ignored by sockserv and a warning is printed.
The configure script works now with Solaris 8 and older.
A bug was fixed in rules.pl where an escaped : is part of a signature message.
The databas connection can now be done via TCP instead of an unix socket. (This can make sense with chrooted programs.) In this case the SocketName must be of the form hostname:port.
The contrib-directory has now two scripts for setting up the database scheme 107 with support for tagged packets and storing the full payload in the database. With this setup you can recreate full pcap files.
getpacket has now the option -z to disable the following of tagged packets, some further checks are added.
The alert and drop facility can be disabled by using the socket name NULL.
The option -q or with servsock the key DropQuiet are added. This disables the writing of of dropped alerts to the unix socket. Note: Dropping is still possible.
Added patch for snort-2.6.0-beta
Some checks were added to the ouput plugin for snort (only part of the patches for snort-2.4.3 and snort-2.6.0).
Finally, a QUICKSTART file is now included.

Changes in version 1.5.0

The schema vseq=107 is supported: we can store sig_gid, the generator id of the event, in the database. Hopefully Base will use it the same way as announced...
Added control thread so some parameters like debug level can be adjusted during run time.
Restriction of one snort process per sensor is removed, now several instances of sockserv can run on one sensor. The sockserv option -N is used to set the sensor name for the database. The length is limited to 63 bytes.
The use of stunnel or ssh tunnel for encrypting the traffic is now possible.
The configure script tries to get the database compile settings via mysqlconfig and/or pgconfig
The snort option -Y disables the default output plugins, so the alerts are only written to the activated output plugins in snort.conf. The default was to write the alerts on disk. Thus we do not need a disk at all on the sensor.
getpacket is now able to recreate a pcap file on 64 bit systems. (Although a mixture of 32/64 bit systems will not work.)
If servsock is terminated via SIGINT or SIGTERM then all cached alerts are written to the swap files.
Cleanup of database access code.
Several bug fixes.
If logging to syslog is used then the PID is added to each line.

Changes in version 1.4.1

added -Z option to servsock to change the timezone to local one (-Z 0) instead of UTC.
corrected bug in Database() if transaction was disabled (reported by Dominik, thanks!)
added offset to event_reference to be unique among restarts of snort. This requires the setting of Reference = 2 in servsock.conf.
corrected bug if last_cid has to be adjusted on a restart of servsock.
added -I flag to getpacket to print some informations of the packet like MAC addresses.
added -V flag to getpacket to print vendor informations based on the (ethernet) MAC addresses. The vendor list is taken from ettercap.
configure checks now if the snort sources are patched.

Changes in 1.4.0 since version 1.3.0

getpacket is now able to rebuild a stream of tagged packets (Many thanks to Alex Butcher for the help to realize this.)
rules.pl was adjusted to deal with rules without given classification or priority.
The receive and send buffer for the unix sockets are set to at least an useful size. With a low RAM installed it could happen that the send buffer was too small to work with a large rebuild TCP packet. (Note: With stream4 enabled this packets could have a size of about 64kB!)
Adjustment of some buffer sizes. On heavy load during a SIGINT/SIGTERM not all threads where terminated
With recent versions of snort (2.1.3 or 2.2.0RC1) it is now possible to write even the log packets to the unix socket. So only the log packets, only the alert packets or both can be inserted in the database.
If both events (log and alert) are activated then the log events are only written to the socket if not already an alert of this event happened.
Additionally snort got a -Q command line option: If this option is present then nothing is written to /var/log/snort. Therefore an alert_null output plugin was added (based on log_null which already existed). But this option is only part of the patches for recent snort versions (2.1.3 and 2.2.0RC1).

Changes between 1.0 and 1.2.3:

A handshake is implemented between sockserv and servsock. This is used to inform sockserv if there are any problems like a missing database or old data from a previous connection not inserted yet.
A swap file feature exists with servsock: If there is a problem with the database the buffered alerts are stored in a swap file and the connections to the remote sensors are closed. If a reconnect happens and the database is available again then first the swap file is read in and then the reconnect of the sensor is accepted.
FLoP runs at least on Linux, Solaris, OpenBSD and FreeBSD.
The use of libbind is now optional.
With a slight extension of the database the full payload can be stored in the database and a pcap file can be rebuild with the program getpacket. This pcap file can be analyzed with programs like Wireshark.
Debug code is compiled in by default and can be activated via command line or the configuration file (if one is available). This can be disabled during configure with the option --disable-debug.
A perl script (contrib/rules.pl) is able to insert the signatures, references, revisions and priority based on the rule files into the database together with the contents of the file classification.config.
On servsock the sensor name is printed during output of statistics in additon to the PID
On servsock a negative UnixSocketPriority reverts the scheme, a value of -2 only alerts on files with priority 1 or 2.
On servsock the sensor name is printed during output of statistics in addition to the PID, received alerts, inserted alerts, dropped alerts and from a swap file restored alerts.

Readme files

There are now a lot of README files available regarding several aspects of the project:

README: The main readme file of the project.
README.barnyard: This file shows some differences between barnyard/mudpit and FLoP.
README.debug: Informations how to activate and use the debug facility.
README.endian: Some comments about the endianess of the computers involved in the project.
README.payload: This file describes how the database scheme has to be extended to store the whole payload of an alert packet. Further here you can find how to activate this feature in servsock.
README.snort: Some limitations in the use of FLoP with snort.
README.ctrl: How to use the control thread via a named pipe.
contrib/README: This is a survey of the useful files in the contrib directory.
contrib/README.rules.pl: This describes the usage of rules.pl to add all signatures to the database.
INSTALL: A short description how to configure, compile and install FLoP.
QUICKSTART: A short description how to set up all for the impatient.

This project uses a modified unix domain socket output plugin of the network intrusion detection system snort. The alerts generated by snort are read from the unix domain socket by another process called sockserv. This process reads from a socket and sends the alerts via TCP to a central server. On the central server a program called servsock reads these data and writes them via an unix domain socket to a database.

So this project is developed for environments with several remote sensors and one central server gathering the informations. With the normal database output plugin there would be several SELECT and INSERT statements via the network which would slow down the INSERT rate. Additionally in this scheme snort is blocked until all data is spooled to the database. The only reason why you will not loose traffic in between is the buffering in the kernel and/or libpcap. If this results in a buffer overrun in the kernel buffer you will not even notice it. In contrast the libpcap will report the dropped packets if it drops them. (With Linux you should use the libpcap version of Phil Woods at http://public.lanl.gov/cpw/ to get useful statistics.)

The advantage of this method is the complete decoupling of the output processing from snort. The programs sockserv and servsock buffer all alerts in cases of a slow network or a slow database access (or a heavy attack is going on generating a lot of alerts in a very short period). Both programs use two threads, one to receive data and one to forward this data either to the central server or to the database.

In addition to avoid an overrun of the internal buffer of the programs there exists a drop feature. Herein the alerts were dropped before they were forwarded to the central server (sockserv) or the database (servsock). A short description of each dropped alert can be e-mailed to a list of recipients.

Finally there is an alert feature which is able to send alerts as e-mails if the priority reaches a given level. This feature is intended to inform an admin on a high level alert. (There is still a problem in the definition of a high level: Is this a high priority value or a low one? This seemed to be changed sometimes between snort-1.8 and snort-2.0.)

Two further programs/features were added:

fpg: A false positive generator to test the whole system under a defined load.
An output option for snort to write statistics to a unix domain socket. This way statistics can be written via a perl script stats.pl to a RRD database.

Documentation

There exist some extended documentation.

There are also some manual pages available (also part of the documentation).

Download

You can download the actual sources here: FLoP-1.6.0

Maybe I will create some binary packages in the future...

Linux (x86) binaries are no longer available. The reason is the usage of the glibc-2.3:

warning: Using 'gethostbyname' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking

The reason seems to be that the glibc still needs the dynamic library to use dlopen. But this is needed at least for host name resolving which can use DNS, files, NIS or something else for name resolution. But these methods are defined on startup and not during compilation.

So maybe I will create statically linked files again...

It will run fine on Linux and Solaris. *BSD (except BSD/OS from WindRiver) should work but I did not find the time to test it in detail. Note: Signals are mainly used for statistics and for a clean exit. Here a clean exit means that all buffered alerts are processed before the program exits.

All other operating systems may work as long as they run a unix system.

Problems will arise on a mixture of 32 bit and 64 bit systems. (One reason is the different size of time_t which is used in the pcap format.)

Problems will arise on a mixture of big and little endian systems. Maybe this will be fixed in the future but actually I have neither the time nor do I have any need for this... Starting with version 1.0.1 any connection from a remote sensor with a different endianess as the central server is now rejected.

Summary

Advantages of FLoP are:

Decoupling of the output from snort. Snort can work on new packets

instead of processing the output.

Buffering of alerts on the sensor. This is useful if you have a shortage on your network to the central server or the servsock process on the central server is not running (maybe it will be restarted due to a change to a newer version...)
Buffering of alerts on the central server. It is not uncommon that the database (especially MySQL) is hanging during a high input rate or the rate is faster than the database is able to store.
Fast writing to the database via an unix domain socket.
E-Mail alerting on high priority alerts.
Drop feature for the worst case. At least the basic alert informations are still available either via E-Mail or on stdout/syslog.
Since version 1.0.6 the alerts which should be dropped on the central server if servsock exits are written to a swap file. So this data is still available.
If alerts have to been dropped because the high water mark was reached then these data are not written to the swap file.

There are also some limitations/disadvantages:

If you use bpf rules to pre-filter snort traffic then this will not be marked in the database. If you see the alert packet in the database then all you know is the name of the sensor and not the actual configuration. (Otherwise you would get a new sensor entry for each sensor with a changed bpf configuration...)
Some informations in the database are missing or have to be inserted by another method, for example the classification. This can be easily done with a simple perl script. With version 1.2.2 there is a perl script in the contrib directory called rules.pl which is able to insert these missing parts.
The traffic between the sensors and the central server is not encrypted. Since our focus is on fast logging we expect that you use a separate network for the communication between the sensors and the central server. (Additionally we assume snort processes running in stealth mode. Therefore a separate network is highly recommended.) Since version 1.5.0 it is possible to use encrypted tunnels like with stunnel or sshd.
Actually only the databases MySQL and PostgreSQL are supported. I prefer to use the PostgreSQL database which is since version 7.3 not really slower as MySQL but uses less memory
Maybe you will find some further limitations...

License

The project is set under the GPL Version 2:

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.

GNU GENERAL PUBLIC LICENSE Version 2, June 1991

Finally

If you have any suggestions, comments, errors,...: Feel free to send me an E-Mail: Dirk

This document was generated using AFT v5.096