Working with Syslog

ClickHouse DB Installation for Nectus Netflow & Syslog Storage

, ,

ClickHouse Database Installation for Nectus Netflow & Syslog Storage

Requirements: Ubuntu Server 18.04.2 LTS (with SSH access)

NOTE: Although ClickHouse can be installed on several different flavors of Linux, Ubuntu Server 16.04 & 18.04 are the only supported Linux distributions for Nectus at this point.

More information about installation on other OS’s can be found here: https://clickhouse.yandex/docs/en/getting_started/

Step 1

Import the public key:

apt-get update

sudo apt-key adv –keyserver keyserver.ubuntu.com –recv C8F1E19FE0C56BD4

NOTE: It is recommended to import the public key if it’s a fresh Ubuntu install.

Otherwise you may get the following error when adding the repository:GPG error: http://repo.yandex.ru/clickhouse/deb/stable main/ Release: The following signatures couldn’t be verified because the public key is not available: NO_PUBKEY C8F1E19FE0C56BD4

Optional commands to run:

sudo apt-key adv –keyserver keyserver.ubuntu.com –recv E0C56BD4

Step 2

Create Clickhouse repository:

sudo apt-get install dirmngr

sudo apt-add-repository “deb http://repo.yandex.ru/clickhouse/deb/stable/ main/”

NOTE: Please edit the sources.list file if you receive the following error:

“E: Mailformed entry 55 in list file /etc/apt/sources.list”.

Delete the entry XX and save/exit the file. Perform the update (and upgrade if you wish):

sudo apt-get update

Step 3

Get the packages list for the latest updates and the dependencies:

apt-get update && apt-get upgrade

Step 4

Download and Install the ClickHouse packages:

sudo apt-get install -y clickhouse-server clickhouse-server-common clickhouse-client

sudo apt-get install –allow-unauthenticated clickhouse-server-common clickhouse-client clickhouse-server

Step 5

Start the Clickhouse server as a daemon:

sudo service clickhouse-server start

Step 6

Now that we have installed the ClickHouse, it is time to test:

NOTE: TCP ports 8123 & 9000 must be open.

Start the Client:

clickhouse-client

Step 7

Create the Netflow/Syslog database.

In this example we are creating the netflow database named NETFLOW (database name is arbitrary):

clickhouse-client

create database NETFLOW

Step 8

We will now add the Clickhouse internal user “root” with password “nectus” to the Users.xml file located at: /etc/clickhouse-server/users.xml

NOTE: Paste the snippet of the code below starting at line 31 after “<users>” in the users.xml file. You can use vi or nano text editor to edit the file. WinSCP can also be used to accomplish this task. Use the file change owner command if needed “sudo chown -R xxxx users.xml”, where “xxxx” is the user that will take over the ownership of the file.

<root>

<password>nectus</password>

<networks incl=”networks” replace=”replace”>

<ip>::/0</ip>

</networks>

<profile>default</profile>

<quota>default</quota>

</root>

Save the file and exit.

Step 9

Restart the ClickHouse Server:

clickhouse-server restart

Step 10

Now that the ClickHouse Server has been restarted, we can start the ClickHouse client using the internal user that we created in step 8:

clickhouse-client –user default –password nectus

Step 11

We have completed the ClickHouse installation. This last step requires login to Nectus to finish the Netflow/Syslog integration.

Open to “Nectus Settings -> General Settings -> Netflow Integration” page

Enter the required information and click Test DB Connection (Remote Server IP is the IP address of the Ubuntu/ClickHouse server). The result should be “Test DB Connection OK”

Click “Run Integration Scripts” and finally Save.

 

Step 12

Restart Nectus NetFlow and Syslog Services.

How to Configure Nectus Syslog Collector to use Local Storage

,

How to Configure Nectus Syslog Collector to use Local Storage

  1. To configure Nectus Syslog collector storage settings go to Main Menu

Settings → General Settings → Syslog Settings

  1. Configure Storage parameters according to this example:

“Syslog Remote Server DB Root Password” should be taken from this file:

C:\Program Files\Nectus\Web\Apache24\htdocs\protected\config\database.ini

  1. After Configuration is finished press “Test DB Connection” to test connectivity to DB

  1. After DB connectivity is Tested, Press “Run Integration Scripts” button to create required SQL Tables.
  2. After Integration Scripts has been executed, Restart Syslog collector service in

“Settings → Services Status”

After Syslog Service is Restarted it should be ready to process and store Syslog Traffic.

 

Cascading Syslog Servers

,

Cascading Syslog Servers

Introduction to the Syslog Protocol

Syslog is a protocol that allows systems to send Event Notification Messages through IP networks to Syslog Servers (also known as Event Message Collectors). There the messages can be sorted, searched, and analyzed to monitor the state of individual devices as well as the overall system.

Syslog messages contain both status information and a Severity Level, which ranges from 0 (zero) to 7. Level 0 messages are emergencies. Level 7 messages signify that the sender is in Debug mode. The meanings of Levels 1 through 6 are application dependent.

2. Multiple Syslog Servers – The Traditional Approach

In some situations you might want to add additional Syslog Servers to your system. Traditionally you would do this by configuring each connected device or server to send messages to the Main Syslog Server and to each Secondary Syslog Server. This configuration is shown in the following image:

This works fine if you just have a few devices. But it quickly becomes impractical as the number of connected devices grows. Imagine configuring 1000+ devices to send Syslog messages to one or more additional servers for a special project, then disconnecting them all later.

This makes the traditional approach impractical for large installations.

3. Multiple Syslog Servers – The Cascading Approach

To avoid the problems of the traditional approach, Nectus implements Cascading Syslog Servers. Instead of connecting each device to each Syslog server, you need only connect them to the primary Syslog server. The primary server can then forward copies of the messages to any secondary servers, as shown in the following image:

This approach makes adding and removing secondary Syslog servers simple. However, forwarding every Syslog message does increase the load on the primary Syslog server. You need to carefully monitor the primary server to avoid overloading it.

Nectus recommends you cascade no more than 10 secondary Syslog servers to avoid overloading the primary server.

3.1 Configuring the Nectus Cascading Syslog Servers Solution

Follow these steps to configure Cascading Syslog Servers:

  1. Click Settings in the Nectus Home Screen.
  2. In the Settings menu that appears, hover the cursor over the General Settings option.
  3. Click the Syslog Settings option that appears. Select the Forwarding IP tab in the Syslog Settings dialog that appears.

  1. Click the Add New IPv4 button to open the Add Forwarding IPv4 dialog.

  1. For each secondary Syslog server add the IPv4 Address of the server, the number of the UDP Port the server is listening on, and a Description of the server.

Working with SNMP Traps in Nectus NMS

, ,

One of the key features of Nectus is ability receive SNMP traps.  This allows the operator to quickly detect physical (links failures, flaps) or logical (adjacencies failures, flaps) changes in the network followed by restoration procedures or root cause analysis.
Let’s see how does it work on example of this Layer 2 diagram:

The SNMP traps are located in top menu “Logs” -> “SNMP traps and Syslog”.

More exactly here:

Once the devices starts sending SNMP traps, they appears withing 2-3 minutes in Nectus:

At the time of the writing, the new SNMP traps decoding is added manually by the Nectus support team, but in the future the operator will have the ability to add the traps decoding with no external assistance.
All these traps were sent by R1 and R2 when GigabitEthernet0/1 on R1 was shut down. Because OSPF was running between R1 and R2, disabling the interface lead to OSPF adjacency between the two routers to go down.
As you can see, some OIDs are represented their original format, whereas some OIDs are represented in a more human readable format.
The details of a trap might look like this:

With the above trap details, you can tell that this is “link down” trap.
However, with other traps, you might need additional knowledge to figure what they represent.
Let’s take another example of detailed trap:

This trap was sent by R1 and it is a neighbor state change(.1.3.6.1.2.1.14.16.2.2):

And it can be read like this: Router-ID 192.168.0.2(.1.3.6.1.2.1.14.1.1) declared neighbor 192.168.0.3(.1.3.6.1.2.1.14.10.1.3) with IP address on the interface 10.0.0.14(.1.3.6.1.2.1.14.10.1.1) as Down(.1.3.6.1.2.1.14.10.1.6 with value of 1).
Obviously this is not very intuitive and Nectus should do all the the decoding so that the operator will not go through all the effort to find what each OID means.

In the future releases user will have an option  to add the decoding for new or unknown SNMP traps via GUI.

 

Syslog server functionality in Nectus NMS

,

This post will cover the SysLog server functionality of  Nectus network monitoring software.
As with any modern network monitoring software Nectus has the ability to receive and store the syslog messages from routers, switches or servers.
Syslog messages can be accessed from Top menu “Logs”: Read more

Nectus Syslog and keyword based alerting

,

One of the unique features of Nectus Syslog service is ability to alert users via Email or Text messages not only

on Syslog message Severity level but on specific keywords inside Syslog message. For example you can configure a rule

to alert via email when there is Syslog message with Severity 2+ and there is string “VPC Peer-Link” inside Syslog message body,

limiting your alerts to only syslog messages related to VPC Peerlinks. You can configure multiple keywords with Alerts going to

different recipients, so the Server team receives the Server specific keywords and Network Team receives the Alerts about

those ugly green boxes locked in MDF closets.

Preventing specific devices from sending messages to Syslog DB

,

If you want to prevent specific device from sending messages to Syslog, you can add its IP address

to Syslog Sender Blacklist. All messages from that IP address will be discarded.

 

Adding to Syslog keyword Blacklist

,

If you want to prevent specific Syslog messages from being added to Syslog Database,

you can add a specific keyword to a Syslog blacklist and all syslog messages that contain this keyword will be discarded.

This does not have retroactive effect on messages that are already in DB.