Working with NetFlow Features

ClickHouse DB Installation for Nectus Netflow & Syslog Storage

, ,

ClickHouse Database Installation for Nectus Netflow & Syslog Storage

Requirements: Ubuntu Server 18.04.2 LTS (with SSH access)

NOTE: Although ClickHouse can be installed on several different flavors of Linux, Ubuntu Server 16.04 & 18.04 are the only supported Linux distributions for Nectus at this point.

More information about installation on other OS’s can be found here: https://clickhouse.yandex/docs/en/getting_started/

Step 1

Import the public key:

apt-get update

sudo apt-key adv –keyserver keyserver.ubuntu.com –recv C8F1E19FE0C56BD4

NOTE: It is recommended to import the public key if it’s a fresh Ubuntu install.

Otherwise you may get the following error when adding the repository:GPG error: http://repo.yandex.ru/clickhouse/deb/stable main/ Release: The following signatures couldn’t be verified because the public key is not available: NO_PUBKEY C8F1E19FE0C56BD4

Optional commands to run:

sudo apt-key adv –keyserver keyserver.ubuntu.com –recv E0C56BD4

Step 2

Create Clickhouse repository:

sudo apt-get install dirmngr

sudo apt-add-repository “deb http://repo.yandex.ru/clickhouse/deb/stable/ main/”

NOTE: Please edit the sources.list file if you receive the following error:

“E: Mailformed entry 55 in list file /etc/apt/sources.list”.

Delete the entry XX and save/exit the file. Perform the update (and upgrade if you wish):

sudo apt-get update

Step 3

Get the packages list for the latest updates and the dependencies:

apt-get update && apt-get upgrade

Step 4

Download and Install the ClickHouse packages:

sudo apt-get install -y clickhouse-server clickhouse-server-common clickhouse-client

sudo apt-get install –allow-unauthenticated clickhouse-server-common clickhouse-client clickhouse-server

Step 5

Start the Clickhouse server as a daemon:

sudo service clickhouse-server start

Step 6

Now that we have installed the ClickHouse, it is time to test:

NOTE: TCP ports 8123 & 9000 must be open.

Start the Client:

clickhouse-client

Step 7

Create the Netflow/Syslog database.

In this example we are creating the netflow database named NETFLOW (database name is arbitrary):

clickhouse-client

create database NETFLOW

Step 8

We will now add the Clickhouse internal user “root” with password “nectus” to the Users.xml file located at: /etc/clickhouse-server/users.xml

NOTE: Paste the snippet of the code below starting at line 31 after “<users>” in the users.xml file. You can use vi or nano text editor to edit the file. WinSCP can also be used to accomplish this task. Use the file change owner command if needed “sudo chown -R xxxx users.xml”, where “xxxx” is the user that will take over the ownership of the file.

<root>

<password>nectus</password>

<networks incl=”networks” replace=”replace”>

<ip>::/0</ip>

</networks>

<profile>default</profile>

<quota>default</quota>

</root>

Save the file and exit.

Step 9

Restart the ClickHouse Server:

clickhouse-server restart

Step 10

Now that the ClickHouse Server has been restarted, we can start the ClickHouse client using the internal user that we created in step 8:

clickhouse-client –user default –password nectus

Step 11

We have completed the ClickHouse installation. This last step requires login to Nectus to finish the Netflow/Syslog integration.

Open to “Nectus Settings -> General Settings -> Netflow Integration” page

Enter the required information and click Test DB Connection (Remote Server IP is the IP address of the Ubuntu/ClickHouse server). The result should be “Test DB Connection OK”

Click “Run Integration Scripts” and finally Save.

 

Step 12

Restart Nectus NetFlow and Syslog Services.

How to Configure Nectus NetFlow Collector to use Local Storage

,

How to Configure Nectus NetFlow Collector to use Local Storage

To configure Nectus Netflow collector storage settings go to Main Menu

Settings → General Settings → NetFlow Integration

Configure Storage parameters according to this example:

“NetFlow Remote Server DB Root Password” should be taken from this file:

C:\Program Files\Nectus\Web\Apache24\htdocs\protected\config\database.ini

After Configuration is finished press “Test DB Connection” to test connectivity to DB

After DB connectivity is Tested, Press “Run Integration Scripts” button to create required SQL

Tables.

After Integration Scripts has been executed, Restart NetFlow collector service in

Top menu “Settings → Services Status”

After NetFlow Service is Restarted it should be ready to process NetFlow Traffic and store it in local DB.

 

NetFlow Reports Supported by Nectus

,

Here is the list of NetFlow reports currently supported by Nectus

Top Applications
Top Protocols
Top Source IP
Top Destination IP
Top Source + Destination IP pairs
Top Source BGP AS
Top Destination BGP AS
Top Source + Destination BGP Pairs

 

 

 

Selecting fastest Database Engine for Netflow Storage

,

Well, if you are reading this article you probably know what NetFlow is and how much data it can generate. 100GB of NetFlow records per day is not something totally unusual in NetFlow business.

We tried most of the commercial and open-source existing database engines like Microsoft SQL Server, MongoDB, PostegreSQL, MySQL or even Oracle and were not happy with results.

We tried turning off all the indexing, implemented daily partitions,  decided not to store some less important NetFlow fields  and may be even

upgraded your storage to those fancy M2 SSD and still NetFlow reports took minutes to appear.

If your NetFlow rates are somewhere under 1,000 flows per second you can skip this reading, pick any  DB and it will produce acceptable results.

Problems becomes visible when your flow rates  reaches 2,000 fps and at 10,000 fps just doing INSERT to your tables takes 70% of the time.

10,000 flows per second produces 600,000 database records every minute and only INSERT statement takes around 40 second to process leaving only 20 seconds of

available time  for any reports to be generated. The main problem here is that conventional DB engines are not optimized for storing read-only sequential data such

as time based event logging or NetFlow. Best database engine for Netflow has to be designed with read-only sequential access in mind, no “Delete-Update” functionality is required for NetFlow.

This restriction allows great simplification of DB internal formatting structure  and processing logic. Second DB feature that best suited for NetFlow is reduction of possible Indexes to one.

There is absolutely no value in having indexes for Source/Destination IP addresses or Source/Destination ports as those indexes only benefit single type of NetFlow report

and each index will double your table size on disk. The only Index that is used in all reports is Index by flow time stamp as all NetFlow reports are focused on very specific time frame.

And last but not least DB feature that is required for perfect NetFlow storage is hardware optimization. Allocating each DB thread to a dedicated CPU core  has shown

to increase query processing time by 10x.

When  developing our own NetFlow collector we tried all well-known DB engines with one performing slightly better than others but none of them were able to support the golden standard of 30Kfps.

This was until we met the ClickHouse, open source DB engine developed by Yandex. ClickHouse has a long list of limitations, but those limitations are implemented with

a single purpose to have the fastest logging DB engine available on the market. With a help of ClickHouse Nectus can process 50,000 flow per second in single VM which  is currently

a record among all commercially available NetFlow collectors.

Download your 60-day Trial of the best NetFlow collector.

 

 

 

 

 

Nectus NetFlow collector VM sizing

,

When selecting specs for Nectus VM that will be used for Netflow collection storage we recommend following specifications:

RAM: Amount of RAM has to be not less that Netflow DB size growth in one week.

Example:  Your Netflow  storage grows by 10GB every day. Recommended RAM amount is 70GB

CPU: Number of logical cores has to be not less than number of Netflow senders

Example: You have 16 routers sending Netflow data. Nectus creates one thread for each sender so ideal CPU configuration is with number of logical cores not less than 16.

Storage: We only recommend dedicated SSD based storage with minimum 200,000 IOPS rating.   We had very good experience with PCI-E Intel Optane SSD.

With recommended specs we should be able to support  close to 30Kfps Flow rates

Download best Netflow collector

 

Nectus sets new record in Netflow Collector speed: 50,000 flows per second

,

Today we have achieved a new millstone with our Netflow collector performance:
50,000 flows per second processing speed was reached in Windows Server 2016 VM.
This rate was reached on VM with 64GB RAM and 1TB M2 SSD.
To simulate this flow rate we used 10 isolated instances of Netflow generator from Virtual Console (www.vconsole.com) with each instance generating 5Kf/sec.
There is no Netflow collector on the market that we are aware which can sustain this Netflow packet rate.
This explains why large cloud providers turn to Nectus for their Netflow processing needs when other tools fails to deliver.

Download 60 day trial:     Nectus Download

How to simulate Netflow packets for testing purposes..

,

We offer NetFlow Traffic Generator utility that can send up to 30k flows per second with randomized parameters.

This tool is ideal for anyone who is developing Netflow Collector functionality.

Allows to generate large amount of NetFlow packets for protocol versions: 5, 6, 7, 8, 9 as if they were coming from real routers or switches.

Windows GUI provides control for every single parameter of the NetFlow packets.

Randomizaton of flow data. Support for multiple collectors. Simulation for up to 5000 routers.

Runs on any Windows OS.

Download Netflow Generator

High Performance (30K flows per second) Netflow collector added to Nectus starting from version 1.2.6

,

Following Netflow reports are available for all Nectus Suite users starting from version 1.2.6

  1. Top Flows by Protocols
  2. Top Flows by Application
  3. Top Flows by BGP AS Source + Destination
  4. Top Flows by BGP AS Source
  5. Top Flows by BGP AS Destination
  6. Top Flows by IP Source + Destination
  7. Top Flows by IP Source
  8. Top Flows by IP Destination
  9. Top Flows by Source Countries
  10. Top Flow by Destination Countries

All reports are supplied with  IP Geolocation information. Netflow collector is a licence free component of Nectus suite.

Supported Netflow formats: V5, V9, IPFIX

Max number of flows per second: 30,000

Netflow collector runs on a dedicated VM or standalone server with following recommended specs:

OS: Windows 64 Bit

RAM: 32GB+

HDD: 1TB SSD

Configuring Netflow collector integration on Nectus 1.28

,

Starting from version 1.28 Nectus supports processing of inbound Netflow packets.

To enable Netflow functionality separate standalone Server or VM is required for Storage.

64 bit MySQL Server has to be installed on Netflow storage VM and DB Name, Root credentials and TCP port

for Netflow Storage DB  has to be configured on main Nectus Server under “Settings -> General Settings -> Netflow Integration”

 

Netflow Collector can support up to 30,000 flow per second.

 

 

Netflow collector functionality added to Nectus starting from version 1.2.8

, ,

Netflow collector functionality added to Nectus starting from version 1.2.8