Mar 31 2014

Kippo attack heatmap in seconds using Kibana and Kippo2ElasticSearch

Continuing from my previous post, here is how to create an attack heat map in seconds using the same ElasticSearch + Kibana instance. First of all we have to download Maxmind’s GeoIP database. The general procedure is super easy (no need to do it):

wget -N
gunzip GeoIP.dat.gz

This will output a single GeoIP.dat file which is a binary format with IP to geolocation data mappings which you can query using an API. The Python version of the latter is easily installable via pip (do this):

pip install GeoIP

Bare in mind that you’ll probably get the “clang: error: unknown argument” failure message but fear not; I have written the solution here if you need it:

We then have to modify the script I posted a little bit, in order to save the two letter country code in the JSON documents before indexing them in ElasticSearch. I have actually decided to pursue this project and publish the (poorly written at this stage, serving as an example) code properly. So just get the Kippo2ElasticSearch files from GitHub:

git clone

It includes the GeoIP database, no need to get it yourself. Edit the MySQL and ES values and you’re ready. After importing the data to ElasticSearch, open Kibana and add a new map panel:

kibana kippo map 1 Kippo attack heatmap in seconds using Kibana and Kippo2ElasticSearchAnd voilà, scroll down and you’ll have a heatmap of attacks:

kibana kippo map 2 Kippo attack heatmap in seconds using Kibana and Kippo2ElasticSearchDo you really need more convincing about the prospects of a project combing honeypots with ElasticSearch + Kibana? icon smile Kippo attack heatmap in seconds using Kibana and Kippo2ElasticSearch

Mar 30 2014

Transferring Kippo’s data to ElasticSearch

I have been investigating ElasticSearch and Kibana for some projects lately and I’ve come to appreciate the easiness of using the two pieces of software together for storing and visualizing data.

This will be an introductory post to something bigger, but I just want to throw the idea out there: let’s transfer honeypot data to ElasticSearch and use Kibana for easy visualization and creation of dashboards.

For Kippo, it all starts with the MySQL database. Our first move is to transfer entries from the DB to ElasticSearch. Now, EleasticSearch accepts JSON documents as input, so we’ll have to convert MySQL rows to JSON objects. The second step is to send those JSON objects to ElasitcSearch for indexing.

The obvious table to convert and send to ElasticSearch is the “auth” table which contains login attempts (timestamp, username, password, success, etc). Here is a quick Python script to do just that (you will need pony and pyes):

#!/usr/bin/env python

import pony.orm
import pony.options
import collections
import json
import pyes

mysql_host = 'localhost'
mysql_port = 3306
mysql_user = 'username'
mysql_pass = 'password'
mysql_db = 'database'

es_host = 'localhost'
es_port = 9200

# We need this, otherwise pony returns an error during the SELECT
pony.options.MAX_FETCH_COUNT = 999999

db = pony.orm.Database('mysql', host=mysql_host, port=mysql_port, user=mysql_user, passwd=mysql_pass, db=mysql_db)

with pony.orm.db_session:
    auth_rows ='SELECT * FROM auth')

es = pyes.ES(es_host + ':' + str(es_port))

for auth_row in auth_rows:
    auth_dict = collections.OrderedDict()
    auth_dict['id'] = auth_row[0]
    auth_dict['session'] = auth_row[1]
    auth_dict['success'] = auth_row[2]
    auth_dict['username'] = auth_row[3]
    auth_dict['password'] = auth_row[4]
    auth_dict['timestamp'] = auth_row[5].strftime("%Y-%m-%dT%H:%M:%S")
    auth_json = json.dumps(auth_dict)
    print auth_json
    es.index(auth_json, 'kippo', 'auth')

(a repo for this and similar scripts has been added here: kippo2elasticsearch)

You can then go to Kibana, add a new histogram panel and in seconds (literally) have the following visualization of time based attack summaries:

kibana kippo Transferring Kippos data to ElasticSearch

Another idea is to use an IP-to-country library and include another field in the JSON object that you can then use in Kibana to create a heatmap of attacks, etc. There are generally many possibilities and I would like to gather ideas if you have anything in mind.

As I said this is just an introductory post, I will come back to this idea in the future, publish some proper open source scripts to parse the data and perhaps guides on how to visualize the results with Kibana. Let me know what you think.

Mar 30 2014

Kippo-Graph 0.9.3 released, with new component: “Kippo-IP”

This is the release of a new version of Kippo-Graph, adding a new component: Kippo-IP. Using Kippo-IP you can get a table view of all attacks and inputs by IP address.

Kippo-IP has been developed by s0rtega, so make sure to send him your thanks!

Download: kippo-graph-0.9.3 or clone/pull from GitHub:

MD5 Checksum: 30FDEC6F6F0F75689E776D61616CD18C
SHA-1 Checksum: 6E31D17965E3DEDCAD5A123A2572EE04820E5FC1


Version 0.9.3:
+ Added Kippo-IP: attack details by IP address.


kippo ip screenshot Kippo Graph 0.9.3 released, with new component: Kippo IP

For comments, suggestions, fixes, please use the Kippo-Graph page:

Mar 24 2014

Kippo-Graph 0.9.2, with Kippo-Playlog!

This is the release of a new version of Kippo-Graph, adding a new component: Kippo-Playlog. Now you can play captured honeypot sessions in real time inside Kippo-Graph!

Kippo-Playlog has been developed by CCoffie, so make sure to send him your thanks!

The support is somewhat experimental, so please update Kippo-Graph, test it with your database and let us know if Kippo-Playlog works as suggested.

Download the new version from here: kippo-graph-0.9.2 or clone/pull from Kippo-Graph’s GitHub repo:

As always, here are the checksums of the tar file:

MD5 Checksum: CC3C27DD5BAA2F5AC15DF1E552F9DD05
SHA-1 Checksum: F88DD3EEAEB14B9079AC2182D6A4D8C4457E62E7


Version 0.9.2:
+ Added experimental playlog display.


kippo playlog Kippo Graph 0.9.2, with Kippo Playlog!

For comments, suggestions, fixes, please use the Kippo-Graph page:

Mar 12 2014

Bypassing “clang: error: unknown argument”

Note: I originally wrote the blog post while installing mitmproxy on OS X Maverics, but it is relevant to the “unknown argument” error in general, so keep reading.

Having a Mac laptop is like going on a journey every single day… My latest issue happened today while trying to install mitmproxy. Python’s pip was exiting with the following error:

clang: error: unknown argument: ‘-mno-fused-madd’ [-Wunused-command-line-argument-hard-error-in-future]

Well, it turns out that the latest (5.1) version of Xcode ships with a compiler that treats unknown passed parameters as errors. From the changelog:

The Apple LLVM compiler in Xcode 5.1 treats unrecognized command-line options as errors. This issue has been seen when building both Python native extensions and Ruby Gems, where some invalid compiler options are currently specified.

Projects using invalid compiler options will need to be changed to remove those options. To help ease that transition, the compiler will temporarily accept an option to downgrade the error to a warning:


Note: This option will not be supported in the future.

To workaround this issue, set the ARCHFLAGS environment variable to downgrade the error to a warning. For example, you can install a Python native extension with:

$ ARCHFLAGS=-Wno-error=unused-command-line-argument-hard-error-in-future easy_install ExtensionName

Similarly, you can install a Ruby Gem with:

$ ARCHFLAGS=-Wno-error=unused-command-line-argument-hard-error-in-future gem install GemName 16214764 updated

So, basically you can install mitmproxy (or any other program with a similar error) by:

ARCHFLAGS=-Wno-error=unused-command-line-argument-hard-error-in-future pip install mitmproxy

Bare in mind that you if you need to combine the above with sudo, you will need to add in the beginning of the command and not before “pip”. Otherwise you can also run the above logged in as root user from the start.

Mar 07 2014

Using KeePass on Mac OS X

If you want to run KeePass in Mac OS X like me, you can do it with Mono (described here for example, and also mentioned in the program’s downloads page) but I find it buggy (random exceptions, crashes, etc that can ruin unsaved work). Instead, you can try using KeePassX. KeePassX is actually an old project, a KeePass client for many platforms. I have used it in the past in Linux-based systems but at that time it couldn’t handle v2 databases and actually Mono worked well under Linux so I used the official application. This doesn’t seem to be the case with OS X though.

Newsflash: KeePassX now works under Mac OS X, and it can also manipulate KeePass v2 databases!

The thing is, if you go to the regular KeePassX downloads page you’ll end up with an old version of the application for Mac that doesn’t really work with v2 databases. Instead, you have to get the so-called KeePassX 2.0 version which you can only find through the site’s News page (sigh). Here is a direct link to the latest version: Please keep in mind that this is an alpha version. Still, I found it to work OK, but I would use a cloud based solution with versioning (see Dropbox, Owncloud, etc) to store the database file just in case.

Another option is to use MacPass, a native open source KeePass client for OS X, but this is even more alpha software, so I would recommend against it for the time being. It is being developed quite actively though and looks a promising alternative for the future.


Effective IDS Testing – The OSNIF’s Top 5

Page 1 of 2512345...1020...Last »