Working with NetFlow Features

How to Configure Nectus NetFlow Collector to use Local Storage

,

How to Configure Nectus NetFlow Collector to use Local Storage

To configure Nectus Netflow collector storage settings go to Main Menu

Settings → General Settings → NetFlow Integration

Configure Storage parameters according to this example:

“NetFlow Remote Server DB Root Password” should be taken from this file:

C:\Program Files\Nectus\Web\Apache24\htdocs\protected\config\database.ini

After Configuration is finished press “Test DB Connection” to test connectivity to DB

After DB connectivity is Tested, Press “Run Integration Scripts” button to create required SQL

Tables.

After Integration Scripts has been executed, Restart NetFlow collector service in

Top menu “Settings → Services Status”

After NetFlow Service is Restarted it should be ready to process NetFlow Traffic and store it in local DB.

 

NetFlow Reports Supported by Nectus

,

Here is the list of NetFlow reports currently supported by Nectus

Top Applications
Top Protocols
Top Source IP
Top Destination IP
Top Source + Destination IP pairs
Top Source BGP AS
Top Destination BGP AS
Top Source + Destination BGP Pairs

 

 

 

Selecting fastest Database Engine for Netflow Storage

,

Well, if you are reading this article you probably know what NetFlow is and how much data it can generate. 100GB of NetFlow records per day is not something totally unusual in NetFlow business.

We tried most of the commercial and open-source existing database engines like Microsoft SQL Server, MongoDB, PostegreSQL, MySQL or even Oracle and were not happy with results.

We tried turning off all the indexing, implemented daily partitions,  decided not to store some less important NetFlow fields  and may be even

upgraded your storage to those fancy M2 SSD and still NetFlow reports took minutes to appear.

If your NetFlow rates are somewhere under 1,000 flows per second you can skip this reading, pick any  DB and it will produce acceptable results.

Problems becomes visible when your flow rates  reaches 2,000 fps and at 10,000 fps just doing INSERT to your tables takes 70% of the time.

10,000 flows per second produces 600,000 database records every minute and only INSERT statement takes around 40 second to process leaving only 20 seconds of

available time  for any reports to be generated. The main problem here is that conventional DB engines are not optimized for storing read-only sequential data such

as time based event logging or NetFlow. Best database engine for Netflow has to be designed with read-only sequential access in mind, no “Delete-Update” functionality is required for NetFlow.

This restriction allows great simplification of DB internal formatting structure  and processing logic. Second DB feature that best suited for NetFlow is reduction of possible Indexes to one.

There is absolutely no value in having indexes for Source/Destination IP addresses or Source/Destination ports as those indexes only benefit single type of NetFlow report

and each index will double your table size on disk. The only Index that is used in all reports is Index by flow time stamp as all NetFlow reports are focused on very specific time frame.

And last but not least DB feature that is required for perfect NetFlow storage is hardware optimization. Allocating each DB thread to a dedicated CPU core  has shown

to increase query processing time by 10x.

When  developing our own NetFlow collector we tried all well-known DB engines with one performing slightly better than others but none of them were able to support the golden standard of 30Kfps.

This was until we met the ClickHouse, open source DB engine developed by Yandex. ClickHouse has a long list of limitations, but those limitations are implemented with

a single purpose to have the fastest logging DB engine available on the market. With a help of ClickHouse Nectus can process 50,000 flow per second in single VM which  is currently

a record among all commercially available NetFlow collectors.

Download your 60-day Trial of the best NetFlow collector.

 

 

 

 

 

Nectus NetFlow collector VM sizing

,

When selecting specs for Nectus VM that will be used for Netflow collection storage we recommend following specifications:

RAM: Amount of RAM has to be not less that Netflow DB size growth in one week.

Example:  Your Netflow  storage grows by 10GB every day. Recommended RAM amount is 70GB

CPU: Number of logical cores has to be not less than number of Netflow senders

Example: You have 16 routers sending Netflow data. Nectus creates one thread for each sender so ideal CPU configuration is with number of logical cores not less than 16.

Storage: We only recommend dedicated SSD based storage with minimum 200,000 IOPS rating.   We had very good experience with PCI-E Intel Optane SSD.

With recommended specs we should be able to support  close to 30Kfps Flow rates

Download best Netflow collector

 

Nectus sets new record in Netflow Collector speed: 50,000 flows per second

,

Today we have achieved a new millstone with our Netflow collector performance:
50,000 flows per second processing speed was reached in Windows Server 2016 VM.
This rate was reached on VM with 64GB RAM and 1TB M2 SSD.
To simulate this flow rate we used 10 isolated instances of Netflow generator from Virtual Console (www.vconsole.com) with each instance generating 5Kf/sec.
There is no Netflow collector on the market that we are aware which can sustain this Netflow packet rate.
This explains why large cloud providers turn to Nectus for their Netflow processing needs when other tools fails to deliver.

Download 60 day trial:     Nectus Download

How to simulate Netflow packets for testing purposes..

,

We offer NetFlow Traffic Generator utility that can send up to 30k flows per second with randomized parameters.

This tool is ideal for anyone who is developing Netflow Collector functionality.

Allows to generate large amount of NetFlow packets for protocol versions: 5, 6, 7, 8, 9 as if they were coming from real routers or switches.

Windows GUI provides control for every single parameter of the NetFlow packets.

Randomizaton of flow data. Support for multiple collectors. Simulation for up to 5000 routers.

Runs on any Windows OS.

Download Netflow Generator

High Performance (30K flows per second) Netflow collector added to Nectus starting from version 1.2.6

,

Following Netflow reports are available for all Nectus Suite users starting from version 1.2.6

  1. Top Flows by Protocols
  2. Top Flows by Application
  3. Top Flows by BGP AS Source + Destination
  4. Top Flows by BGP AS Source
  5. Top Flows by BGP AS Destination
  6. Top Flows by IP Source + Destination
  7. Top Flows by IP Source
  8. Top Flows by IP Destination
  9. Top Flows by Source Countries
  10. Top Flow by Destination Countries

All reports are supplied with  IP Geolocation information. Netflow collector is a licence free component of Nectus suite.

Supported Netflow formats: V5, V9, IPFIX

Max number of flows per second: 30,000

Netflow collector runs on a dedicated VM or standalone server with following recommended specs:

OS: Windows 64 Bit

RAM: 32GB+

HDD: 1TB SSD

Configuring Netflow collector integration on Nectus 1.28

,

Starting from version 1.28 Nectus supports processing of inbound Netflow packets.

To enable Netflow functionality separate standalone Server or VM is required for Storage.

64 bit MySQL Server has to be installed on Netflow storage VM and DB Name, Root credentials and TCP port

for Netflow Storage DB  has to be configured on main Nectus Server under “Settings -> General Settings -> Netflow Integration”

 

Netflow Collector can support up to 30,000 flow per second.

 

 

Netflow collector functionality added to Nectus starting from version 1.2.8

, ,

Netflow collector functionality added to Nectus starting from version 1.2.8