1 Comment

Summary:

Today, I read a press release from Sourcefire touting its “big data” approach to security and the fact that its Immunet anti-malware-for-PCs product is now monitoring 2 million endpoints. I reached out to find out what’s under Immunet’s covers and, no surprise, found Hadoop.

No access

Sourcefire is now monitoring 2 million endpoints as part of its Immunet anti-malware product, and Hadoop is doing the heavy lifting of analyzing the hundreds of terabytes of data those endpoints are pumping into the company’s centralized data store. This just goes to prove my point that security and big data are a match made in heaven.

As is the case with many security services, Immunet isn’t a Hadoop product as much as it is a product that uses Hadoop (a sign that the technology is maturing). Two millions endpoints generate a lot of data (in number of items, at least, if not in volume), and it takes some special tools to store and process all that information. The more information that SourceFire can determine about threats, the better it can protect is users. Users don’t care about Hadoop, NoSQL or any other IT buzzwords — they just care that their computers are safe — but companies like Sourcefire certainly do.

According to an email from Zulfikar Ramzan, Sourcefire’s chief scientist within its Cloud Technology Group, “Hadoop is one of the more prominent technologies we use, though we have also built some custom technologies as well.” He said the company also constantly evaluates new technologies such as NoSQL databases, but the trick is finding tools that are “well baked enough for use in production environments” and that meet specific needs. Hadoop, for example, is great for general data mining purposes, but SourceFire has custom-built some tools for real-time workloads that the batch-oriented Hadoop can’t readily address.

As for what it does with all that data — hundreds of terabytes and growing — Ramzan said SourceFire is primarily concerned with detecting anomalies, whitelisting “clean” files and indentifying trends. The first two are pretty self-explanatory — it’s important to detect new threats based on suspicious activity, but it’s also important to not falsely label known safe files as being infected — but the latter is a situation where big data techniques can take traditional security analytics to the next level. Ramzan wrote that, “[W]e mine data for threat related trends — such as which threats are the most popular, what geographic regions are seeing disproportionate threat activity, what global threat characteristics we are seeing, etc.”

Presumably, Sourcefire also utilizes its repository of threat data and analytics tools to power its line of intrustion-detection-and-prevention products designed for business users rather than consumers. ipTrust is already doing just that, using Hadoop, Cassandra and other tools in a system that assigns reputation scores to IP addresses attempting to access a company’s network.

Sometimes, however, Hadoop isn’t the answer even in decidedly big data environments. CloudFlare CEO Matthew Prince recently explained his company, which provides network security and performance for web sites, tried Hadoop early but found it didn’t scale or perform up to the company’s needs. Rather than spend money it didn’t have as an early-stage startup trying to make Hadoop fit its needs, which involve a constant stream of traffic data, CloudFlare built its own tool. The result was “SortaSQL,” a hybrid key-value store that combines the PostgreSQL database with the Kyoto Cabinet key-value store.

But Hadoop or not, the song remains the same: companies building cloud-based security services have data needs that legacy database software can’t handle. They also want to do new things to the data to glean even more insights from it. Given the incredible number of Internet-connected devices on the planet — mobile phones, for example, now outnumber people in the United States– it seems there’s a big business in giving security companies the big data tools they need to do their jobs without making them reinvent the wheel at the data-platform layer.

Image courtesy of Geograph user Ross.

  1. More proof that big data, security are soulmates http://t.co/1HEKkscJ #Cloudcomputing

    Share
  2. More proof that big data, security are soulmates http://t.co/mpRDYDek

    Share
  3. More proof that big data, security are soulmates: Today, I read a press release from Sourcefire touting its “big… http://t.co/hNH2vMwA

    Share
  4. More proof that big data, security are soulmates http://t.co/RvmZOPSM @GigaOM

    Share
  5. More proof that big data, security are soulmates http://t.co/1N0k8nZl

    Share
  6. More proof that big data, security are soulmates http://t.co/VO1B1rEj

    Share
  7. More proof that big data, security are soulmates http://t.co/rLgD3bgW

    Share
  8. More proof that big data, security are soulmates http://t.co/GamMd3OJ

    Share
  9. More proof that big data, security are soulmates http://t.co/vsRiThtK #cloud #gigaom

    Share
  10. More proof that big data, security are soulmates Today, I read a press release from Sourcefire touting i #voip #telcom http://t.co/eQtRa37H

    Share
  11. More proof that big data, security are soulmates http://t.co/JzuT4bem

    Share
  12. More proof that big data, security are soulmates http://t.co/XbZGtltH

    Share
  13. More proof that big data, security are soulmates http://t.co/CdG5JlzV

    Share
  14. More proof that big data, security are soulmates http://t.co/EKUXHJzG

    Share
  15. More proof that big data, security are soulmates http://t.co/FkAiV1pE

    Share
  16. RT @gigaom: More proof that big data, security are soulmates http://t.co/VO1B1rEj

    Share
  17. Interesting piece from @gigaom: More proof that big data, security are soulmates http://t.co/iSPIhTN8

    Share
  18. RT @gigaom: More proof that big data, security are soulmates http://t.co/VO1B1rEj

    Share
  19. RT @gigaom: More proof that big data, security are soulmates http://t.co/VO1B1rEj

    Share
  20. gigaom: More proof that big data, security are soulmates http://t.co/Ga9NchCZ

    Share
  21. More proof that big data, security are soulmates http://t.co/uHUIUAi3 via Derrick Harris

    Share
  22. More proof that big data, security are soulmates http://t.co/cGVlztx1

    Share
  23. RT @gigaom: More proof that big data, security are soulmates http://t.co/MwtCvmQ1

    Share
  24. RT @gigaom: More proof that big data, security are soulmates http://t.co/AOx983bY

    Share
  25. RT @gigaom: More proof that big data, security are soulmates http://t.co/VO1B1rEj

    Share
  26. More proof that big data, security are soulmates: http://t.co/p2Dkwwua

    Share
  27. More proof that big data, security are soulmates: Sourcefire is now monitoring 2 million… http://t.co/ELuIxpCk

    Share
  28. RT @gigaom: More proof that big data, security are soulmates http://t.co/VO1B1rEj

    Share
  29. More proof that big data, security are soulmates http://t.co/s9Bk9j59

    Share
  30. More proof that big data, security are soulmates http://t.co/xKlbgWZP

    Share
  31. Social Media News : More proof that big data, security are soulmates http://t.co/xYS0z3bA

    Share
  32. More proof that big data, security are soulmates — Cloud Computing … http://t.co/LyGJqDkQ

    Share
  33. RT @gigaom: More proof that big data, security are soulmates http://t.co/21BIeVgv

    Share
  34. More proof that big data, security are soulmates: Today, I read a press release from Source… http://t.co/BsmL2XmW CloudComputingTopics

    Share
  35. http://t.co/34ljDy3N More proof that big data, security are soulmates http://t.co/whGOZDgD http://t.co/1YPg7MHz

    Share
  36. http://t.co/mfdFlsOi More proof that big data, security are soulmates http://t.co/kVqFhuGn http://t.co/XVd39lHk

    Share
  37. http://t.co/bxF7daS4 More proof that big data, security are soulmates http://t.co/LiONF7I3 http://t.co/jKdoreJ1

    Share
  38. http://t.co/mfJSrIJx More proof that big data, security are soulmates http://t.co/MitZ641c http://t.co/AaChyp6t

    Share
  39. http://t.co/KEI0cI2Y More proof that big data, security are soulmates http://t.co/pb3v5zPw http://t.co/5f2ioHda

    Share
  40. http://t.co/ANwJzoQm More proof that big data, security are soulmates http://t.co/543zWwGv http://t.co/ccbejud4

    Share
  41. http://t.co/l5BXcnjJ More proof that big data, security are soulmates http://t.co/L3Fp2pin http://t.co/ZZEKD0oS

    Share
  42. http://t.co/RhmFF7Dv More proof that big data, security are soulmates http://t.co/yHoBvo6Y http://t.co/xstHGGQW

    Share
  43. http://t.co/qM7wNnAA More proof that big data, security are soulmates http://t.co/uYJYt26q http://t.co/fNc1tERJ

    Share
  44. http://t.co/Py8dWpyB More proof that big data, security are soulmates http://t.co/6U8OY7aB http://t.co/fwX7KSN5

    Share
  45. http://t.co/pWy6lGxr More proof that big data, security are soulmates http://t.co/UIpOtcK6 http://t.co/slvTVo4s

    Share
  46. http://t.co/KTvHXFtV More proof that big data, security are soulmates http://t.co/Ifr1bJxW http://t.co/a5jO09MD

    Share
  47. http://t.co/BioexqUw More proof that big data, security are soulmates http://t.co/JgMGcMZz http://t.co/fGGPxlWC

    Share
  48. More proof that big data, security are soulmates: The first two are pretty self-explanatory — it’s important to … http://t.co/NASxDSlA

    Share
  49. #cloud More proof that big data, security are soulmates – Today, I read a press release from Sourcefire touting its … http://t.co/WhOI9wMl

    Share
  50. RT @Alltop_Social Alltop_Social: More proof that big data, security are soulmates http://t.co/ASYiRT7o Social-Media…. http://t.co/IAw8LPLo

    Share
  51. RT @gigaom: More proof that big data, security are soulmates http://t.co/VO1B1rEj

    Share
  52. RT @gigaom: More proof that big data, security are soulmates http://t.co/VO1B1rEj

    Share
  53. RT @gigaom: More proof that big data, security are soulmates http://t.co/VO1B1rEj

    Share
  54. More proof that big data, security are soulmates: Today, I read a press release from Sourcefire touting its “big… http://t.co/6EqjSih5

    Share
  55. More proof that big data, security are soulmates: According to an email from Zulfikar Ramzan, So… http://t.co/rlD3gUfM #bigdata #blogs

    Share
  56. RT @gigaom: More proof that big data, security are soulmates http://t.co/VO1B1rEj

    Share
  57. RT @gigaom: More proof that big data, security are soulmates http://t.co/VO1B1rEj

    Share
  58. More proof that big data, security are soulmates http://t.co/Z5lGOux4

    Share
  59. More proof that big data, security are soulmates http://t.co/P2nZicX4

    Share
  60. More proof that big data, security are soulmates http://t.co/qJxyPYbE via @zite

    Share
  61. More proof that big data, security are soulmates http://t.co/YKnE1qlz

    Share
  62. RT @gigaom: More proof that big data, security are soulmates http://t.co/VO1B1rEj

    Share
  63. : More proof that big data, security are soulmates http://t.co/5VjYErqG

    Share
  64. Nice article about Sourcefire’s “big data” backend that our Immunet technology uses.
    http://t.co/g27jJKn1

    Share
  65. RT @mroesch: Nice article about Sourcefire’s “big data” backend that our Immunet technology uses.
    http://t.co/g27jJKn1

    Share
  66. RT @mroesch: Nice article about Sourcefire’s “big data” backend that our Immunet technology uses.
    http://t.co/g27jJKn1

    Share
  67. More proof that big data, security are soulmates http://t.co/MVhA295f

    Share
  68. #Security is a great #BigData app: More proof that big data & security are soulmates http://t.co/8BpaW7U0 v @sheynkman #Hadoop #NoSQL

    Share
  69. RT @mroesch: Nice article about Sourcefire’s “big data” backend that our Immunet technology uses.
    http://t.co/g27jJKn1

    Share
  70. RT @mroesch: Nice article about Sourcefire’s “big data” backend that our Immunet technology uses.
    http://t.co/g27jJKn1

    Share
  71. RT @gigaom: More proof that big data, security are soulmates http://t.co/VO1B1rEj

    Share
  72. More proof that big data, security are soulmates — Cloud Computing …: Today, I read a press re… http://t.co/CtC70qq5 #bigdata #blogs

    Share
  73. The big data game. It is not about Hadoop a product but rather how you use it to fit needs. http://t.co/gr3pjJzX

    Share
  74. “@JennaHannon: The big data game. It is not about Hadoop a product but rather how you use it to fit needs. http://t.co/B0AiQEnx” Agree 100%!

    Share
  75. Add #m2m endpoints and this is even more so. More proof that big data, security are soulmates http://t.co/2xpY3NpC

    Share
  76. RT @campione: #Security is a great #BigData app: More proof that big data & security are soulmates http://t.co/8BpaW7U0 v @sheynkman #Hadoop #NoSQL

    Share
  77. RT @gigaom: More proof that big data, security are soulmates http://t.co/tHJ0O1Zn

    Share
  78. More proof that big data, security are soulmates http://t.co/vh6lmBlZ

    Share
  79. More proof that big data, security are soulmates http://t.co/Oq2mHijI

    Share
  80. RT @gigaom: More proof that big data, security are soulmates http://t.co/VO1B1rEj

    Share
  81. More proof that big data, security are soulmates http://t.co/mbur5xWI

    Share
  82. More proof that big data, security are soulmates http://t.co/y4JCLZFK

    Share
  83. More proof that big data, security are soulmates http://t.co/GcduZ0QN Very interested to see how big data and security evolve in 2012.

    Share
  84. More proof that big data, security are soulmates — Cloud Computing News http://t.co/yYZcNNAd, see more http://t.co/NEyVVHnO

    Share
  85. Important: “More proof that big data, security are soulmates” http://t.co/rfgR7D5Z

    Share
  86. RT @gigaom: More proof that big data, security are soulmates http://t.co/VO1B1rEj

    Share
  87. A pattern for aggregating security data
    RT @gigaom: More proof that big data, security are soulmates http://t.co/OxQ74izE #BigData

    Share
  88. More proof that big data, security are soulmates http://t.co/7FTA2EzW via @zite

    Share
  89. Big data, big security. Everything from log file analysis to malware detection is a big data problem. http://t.co/mpqe5xKl

    Share
  90. More proof that big data, security are soulmates http://t.co/ukb2TzD6

    Share
  91. More proof that big data, security are soulmates http://t.co/D5KJqNAP via @zite

    Share
  92. More proof that big data, security are soulmates: http://t.co/2SnBkqqO #cloudcomputing

    Share
  93. More proof that big data, security are soulmates http://t.co/oAHxoHNi

    Share
  94. More proof that big data, security are soulmates http://t.co/dT4FXkFY

    Share
  95. RT @mikeloukides: Big data, big security. Everything from log file analysis to malware detection is big data problem. http://t.co/m1fER37Q

    Share
  96. RT @gigaom: More proof that big data, security are soulmates http://t.co/VO1B1rEj

    Share
  97. RT @gigaom More proof that big data, security are soulmates http://t.co/hLEk4soD <– will security benefit from big data? #bigdata #hadoop

    Share
  98. RT @Sourcefire: RT @gigaom More proof that big data, security are soulmates http://t.co/hLEk4soD <– will security benefit from big data? #bigdata #hadoop

    Share
  99. RT @Sourcefire: RT @gigaom More proof that big data, security are soulmates http://t.co/hLEk4soD <– will security benefit from big data? #bigdata #hadoop

    Share
  100. #networksecurity More proof that big data, security are soulmates — Cloud Computing … – Today, I read a press r… http://t.co/VKuHrCaU

    Share
  101. RT @Sourcefire: RT @gigaom More proof that big data, security are soulmates http://t.co/hLEk4soD <– will security benefit from big data? #bigdata #hadoop

    Share
  102. Derrick, regarding your view on “finding tools that are well baked enough for use in production environments,” I believe that HPCC is well baked and production ready. It is also mature and fully enterprise ready. Visit http://hpccsystems.com for more information.

    Share
  103. RT @elasticsecurity: More proof that big data, security are soulmates: http://t.co/2SnBkqqO #cloudcomputing

    Share
  104. More proof that big data, security are soulmates http://t.co/ueOweE6c #cloud #saas

    Share
  105. More proof that #BigData, #security are soulmates http://t.co/NaAkXGYx via @gigaom #in

    Share
  106. More proof that #BigData, #security are soulmates. http://t.co/QcMsDYAN

    Share
  107. More proof that big #data, #security are soulmates: http://t.co/PjsuRwcQ (via @gigaom)

    Share
  108. More proof that big data, security are soulmates http://t.co/0Tazcpmn < so true. Not mentioned, but relevant are netflow and pcap datasets.

    Share
  109. Big data and security team up to form an undisputed power couple @gigaom http://t.co/C04ynERa

    Share
  110. RT @privatewifi: More proof that big #data, #security are soulmates: http://t.co/PjsuRwcQ (via @gigaom)

    Share
  111. More proof that big data, security are soulmates http://t.co/5VjYErqG

    Share

Comments have been disabled for this post