oinkception

A project I’m involved with that’s still in it’s early stages is the development of an automated, intelligent security environment that consists of:

  • Firewall, router and switch configuration management (existing in-house developed software).
  • Asset management and vulnerability scanning (Metasploit and OpenVAS).
  • Malware detection and analysis (analysis performed by Cuckoo).
  • Intrusion detection and analysis.

The last two are the final pieces to our puzzle, and given the data has high:

  • Variety: applicance, application and server logs.
  • Velocity and Volume: dozens of networks across the globe, hundrens of servers, 1000+ databases, 1000+ applications.

… it’s really a no brainer to use Hadoop as the storage framework. To dip our toes into the newfound waters of big data, whilst also evaluating a solution that could prove useful in piecing together our environment, I pieced together a network analysis server consisting of Snort, Hadoop, Pig and PacketPig. Continue reading “oinkception”