Working with Syslog

ClickHouse DB Installation for Nectus Netflow & Syslog Storage

, ,

Requirements:  Operating System: Ubuntu 22.04.1 LTS       RAM: 16GB   vCPU: 2

More information about installation: https://clickhouse.com/docs/en/install/#self-managed-install

Step 1

Open terminal and install required packages:

sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates dirmngr

Step 2

Import the ClickHouse public key:

sudo apt-key adv –keyserver hkp://keyserver.ubuntu.com:80 –recv 8919F6BD2B48D754

NOTE: It is recommended to import the public key if it’s a fresh Ubuntu install.

Otherwise, you may get the following error when adding the repository:

GPG error: http://repo.clickhouse.com/deb/stable/main/release:
The following signatures couldn’t be verified because the public key is not available:

NO_PUBKEY 8919F6BD2B48D754

Step 3

Add ClickHouse repository:

echo “deb https://packages.clickhouse.com/deb stable main” | sudo tee \
/etc/apt/sources.list.d/clickhouse.list

Step 4

Install ClickHouse:

NOTE: During the installation you will be asked to create a password for default user.

Generate a strong password and save it, it will be used for further configuration.

sudo apt-get update
sudo apt-get install -y clickhouse-server clickhouse-client

Step 5

Start ClickHouse server as a daemon:

sudo service clickhouse-server start

Step 6

Now after installation it is time to test connection to ClickHouse database.

Start client (use password generated on Step 4):

clickhouse-client –password

NOTE: Use Ctrl+D to exit ClickHouse client.
NOTE: TCP ports 8123 and 9000 must be open

Step 7

NOTE: Use Ctrl+D to exit ClickHouse client.

While being connected to the ClickHouse, create required Netflow & Syslog databases:

CREATE DATABASE NETFLOW;
CREATE DATABASE SYSLOG;

ClickHouse Configuration

Step 8

By default, ClickHouse allows inbound connections from localhost only.

To allow connections from remote IP-s

edit /etc/clickhouse-server/config.xml:

sudo vim /etc/clickhouse-server/config.xml

Find <listen_host> parameters (one for IPv6 and one for IPv4) allowing all connections after <logger> section, and uncomment them.

Also uncomment and change to 1 <listen_reuse_port> tag for allowing simultaneous IPv4 and IPv6 interfaces:

<listen_host>::</listen_host>

<!– Same for hosts without support for IPv6: –>
<listen_host>0.0.0.0</listen_host>

<listen_reuse_port>1</listen_reuse_port>

Step 9

Restart ClickHouse to apply new parameters:

sudo service clickhouse-server restart

Step 10

Add internal ClickHouse user “root” for Nectus server to connect to DB.

For that edit users.xml file:

sudo vi /etc/clickhouse-server/users.xml

Find <users> section right after <profiles> section.

Add code snippet with user configuration from below, right between </users> and <default> tags, as on a screenshot below.

Generate strong password, replace it in snippet below, and then save file:

<root>
<password>**********</password>
<networks incl=”networks” replace=”replace”>
<ip>::/0</ip>
</networks>
<profile>default</profile>
<quota>default</quota>
</root>

NOTE: If file was overwritten by WinSCP or any other similar tool, verify that file has correct ownership:

sudo chown clickhouse:clickhouse /etc/clickhouse-server/users.xml

Step 11

Restart ClickHouse to apply new user:

sudo service clickhouse-server restart

Step 12

Verify newly created user able to log in:

clickhouse-client -u root –password

Step 13

Disable limit of max database size to drop. By default, ClickHouse doesn’t allow to drop table bigger than 50 Gb to prevent occasional data loss.

NOTE: More info here: https://cloud.yandex.com/en/docs/managed-clickhouse/concepts/settings-list#dbms-cluster-settings

To disable it, edit config file /etc/clickhouse-server/config.xml:

sudo vi /etc/clickhouse-server/config.xml

Find and uncomment line to disable limit of max table size to drop:

<max_table_size_to_drop>0</max_table_size_to_drop>

Restart ClickHouse to apply change:

sudo service clickhouse-server restart

Nectus Configuration

Step 14

We have completed the ClickHouse installation. This last step requires login to Nectus to finish the NetFlow/Syslog integration.

Open to “Nectus Settings -> General Settings -> Netflow Integration” page:

Enter the required information and click Test DB Connection (Remote Server IP is the IP address of the Ubuntu/ClickHouse server). The result should be “Test DB Connection OK”

Click “Run Integration Scripts” and finally Save.

Step 15

Restart Nectus NetFlow and Syslog Services.

How to Configure Nectus Syslog Collector to use Local Storage

,

How to Configure Nectus Syslog Collector to use Local Storage

  1. To configure Nectus Syslog collector storage settings go to Main Menu

Settings → General Settings → Syslog Settings

  1. Configure Storage parameters according to this example:

“Syslog Remote Server DB Root Password” should be taken from this file:

C:\Program Files\Nectus\Web\Apache24\htdocs\protected\config\database.ini

  1. After Configuration is finished press “Test DB Connection” to test connectivity to DB

  1. After DB connectivity is Tested, Press “Run Integration Scripts” button to create required SQL Tables.
  2. After Integration Scripts has been executed, Restart Syslog collector service in

“Settings → Services Status”

After Syslog Service is Restarted it should be ready to process and store Syslog Traffic.

 

Cascading Syslog Servers

,

Cascading Syslog Servers

Introduction to the Syslog Protocol

Syslog is a protocol that allows systems to send Event Notification Messages through IP networks to Syslog Servers (also known as Event Message Collectors). There the messages can be sorted, searched, and analyzed to monitor the state of individual devices as well as the overall system.

Syslog messages contain both status information and a Severity Level, which ranges from 0 (zero) to 7. Level 0 messages are emergencies. Level 7 messages signify that the sender is in Debug mode. The meanings of Levels 1 through 6 are application dependent.

2. Multiple Syslog Servers – The Traditional Approach

In some situations you might want to add additional Syslog Servers to your system. Traditionally you would do this by configuring each connected device or server to send messages to the Main Syslog Server and to each Secondary Syslog Server. This configuration is shown in the following image:

This works fine if you just have a few devices. But it quickly becomes impractical as the number of connected devices grows. Imagine configuring 1000+ devices to send Syslog messages to one or more additional servers for a special project, then disconnecting them all later.

This makes the traditional approach impractical for large installations.

3. Multiple Syslog Servers – The Cascading Approach

To avoid the problems of the traditional approach, Nectus implements Cascading Syslog Servers. Instead of connecting each device to each Syslog server, you need only connect them to the primary Syslog server. The primary server can then forward copies of the messages to any secondary servers, as shown in the following image:

This approach makes adding and removing secondary Syslog servers simple. However, forwarding every Syslog message does increase the load on the primary Syslog server. You need to carefully monitor the primary server to avoid overloading it.

Nectus recommends you cascade no more than 10 secondary Syslog servers to avoid overloading the primary server.

3.1 Configuring the Nectus Cascading Syslog Servers Solution

Follow these steps to configure Cascading Syslog Servers:

  1. Click Settings in the Nectus Home Screen.
  2. In the Settings menu that appears, hover the cursor over the General Settings option.
  3. Click the Syslog Settings option that appears. Select the Forwarding IP tab in the Syslog Settings dialog that appears.

  1. Click the Add New IPv4 button to open the Add Forwarding IPv4 dialog.

  1. For each secondary Syslog server add the IPv4 Address of the server, the number of the UDP Port the server is listening on, and a Description of the server.

Working with SNMP Traps in Nectus NMS

, ,

One of the key features of Nectus is ability receive SNMP traps.  This allows the operator to quickly detect physical (links failures, flaps) or logical (adjacencies failures, flaps) changes in the network followed by restoration procedures or root cause analysis.
Let’s see how does it work on example of this Layer 2 diagram:

The SNMP traps are located in top menu “Logs” -> “SNMP traps and Syslog”.

More exactly here:

Once the devices starts sending SNMP traps, they appears withing 2-3 minutes in Nectus:

At the time of the writing, the new SNMP traps decoding is added manually by the Nectus support team, but in the future the operator will have the ability to add the traps decoding with no external assistance.
All these traps were sent by R1 and R2 when GigabitEthernet0/1 on R1 was shut down. Because OSPF was running between R1 and R2, disabling the interface lead to OSPF adjacency between the two routers to go down.
As you can see, some OIDs are represented their original format, whereas some OIDs are represented in a more human readable format.
The details of a trap might look like this:

With the above trap details, you can tell that this is “link down” trap.
However, with other traps, you might need additional knowledge to figure what they represent.
Let’s take another example of detailed trap:

This trap was sent by R1 and it is a neighbor state change(.1.3.6.1.2.1.14.16.2.2):

And it can be read like this: Router-ID 192.168.0.2(.1.3.6.1.2.1.14.1.1) declared neighbor 192.168.0.3(.1.3.6.1.2.1.14.10.1.3) with IP address on the interface 10.0.0.14(.1.3.6.1.2.1.14.10.1.1) as Down(.1.3.6.1.2.1.14.10.1.6 with value of 1).
Obviously this is not very intuitive and Nectus should do all the the decoding so that the operator will not go through all the effort to find what each OID means.

In the future releases user will have an option  to add the decoding for new or unknown SNMP traps via GUI.

 

Syslog server functionality in Nectus NMS

,

This post will cover the SysLog server functionality of  Nectus network monitoring software.
As with any modern network monitoring software Nectus has the ability to receive and store the syslog messages from routers, switches or servers.
Syslog messages can be accessed from Top menu “Logs”: Read more

Nectus Syslog and keyword based alerting

,

One of the unique features of Nectus Syslog service is ability to alert users via Email or Text messages not only

on Syslog message Severity level but on specific keywords inside Syslog message. For example you can configure a rule

to alert via email when there is Syslog message with Severity 2+ and there is string “VPC Peer-Link” inside Syslog message body,

limiting your alerts to only syslog messages related to VPC Peerlinks. You can configure multiple keywords with Alerts going to

different recipients, so the Server team receives the Server specific keywords and Network Team receives the Alerts about

those ugly green boxes locked in MDF closets.

Preventing specific devices from sending messages to Syslog DB

,

If you want to prevent specific device from sending messages to Syslog, you can add its IP address

to Syslog Sender Blacklist. All messages from that IP address will be discarded.

 

Adding to Syslog keyword Blacklist

,

If you want to prevent specific Syslog messages from being added to Syslog Database,

you can add a specific keyword to a Syslog blacklist and all syslog messages that contain this keyword will be discarded.

This does not have retroactive effect on messages that are already in DB.