Linux Monitoring Agent Installation

Linux Monitoring Agent Installation

Nectus Linux Monitoring Agent is required to be installed on target machine to enable Nectus to monitor Linux system parameters such as CPU and RAM utilization, Storage and Network Utilization.

Agent collects all the required monitoring metrics and reports to the main Nectus Server every 5 minutes.

 

Step 1: Download Monitoring Agent Package

Download binary package appropriate to your target machine Linux flavor and version from the download page.

 

Step 2a: Installation for Debian-based systems (including Ubuntu)

After downloading the package run the following installation command:

$ sudo dpkg -i <package-name>

This will install Nectus monitoring agent on your target machine. If everything is OK you’ll see some output like this:

$ sudo dpkg -i nectus-agent-1.0-1.deb

(Reading database … 79643 files and directories currently installed.)

Preparing to unpack nectus-agent-1.0-1.deb …

Unpacking nectus-agent (1.0-1) over (1.0-1) …

Setting up nectus-agent (1.0-1) …

Processing triggers for systemd (229-4ubuntu21.21) …

Processing triggers for ureadahead (0.100.0-19) …

Step 2b: Installation for RedHat and CentOS

After downloading the package run the following installation command:

$ sudo rpm -U <package-name>.rpm

You’ll see output like this:

$ sudo rpm -U nectus-agent-1.0-1.el7.x86_64.rpm

Preparing…

 ################################# [100%]

Updating / installing…

    1:nectus-agent-1.0-1.el7           

################################# [100%]

 

Step 3: Configuration

After installation is complete you need to set up the agent’s basic configuration.

Nectus agent configuration file is located at: /etc/nectus/nectus-agent.conf.

Default configuration file content is listed in Appendix 1. Most of the values are set to defaults and don’t require a change except following two parameters:

  • SERVER_PRESHARED_KEY
  • AGENT_PORT

SERVER_PRESHARED_KEY is used to encrypt messages between the Agent and Nectus Server. Initially it’s set to value <YourSecretKey>.

Edit this parameter to match the key configured in Nectus GUI (Settings->General Settings ->Linux Monitoring Integration).

SERVER_PRESHARED_KEY is a global parameters that is used for all Linux Servers.

AGENT_PORT parameter determines the TCP port used by Agent to listen for incoming connections from the Nectus server. By default the TCP port is set to 5400.

AGENT_PORT is Server specific parameter that can be different on each Linux Server.

 

Step 4: Generate SSL Certificate

The Linux Monitoring Agent listens on pre-configured TCP port for incoming HTTPS connection and needs an SSL certificate to operate.

Default location for storing certificate files is /etc/nectus but it’s configurable (parameter SSL_CERTIFICATE_DIR in the configuration file).

The Nectus agent is shipped with initial self-signed certificate which allows the module to process but it’s strongly recommended to replace this certificate.

You can either use some existing certificate or generate a new self-signed one.

To generate new self-signed certificate run commands:

$ cd /etc/nectus

$ openssl dhparam -out dh.pem 1024

$ openssl req -newkey rsa:2048 -nodes -keyout key.pem -x509 -days 10000 -out certificate.pem

The directory pointed by SSL_CERTIFICATE_DIR parameter should finally contain files dh.pem, key.pem and certificate.pem.

 

Step 5: Open TCP port at the Firewall

If there is firewall between Nectus Server and Linux Monitoring Agent you need to permit connections on TCP port 5400 (Unless it is changed to a different port)

Make sure your Linux internal firewall also permits inbound connections on this port.

 

Step 6: Start Agent

Restart the agent in order for configuration changes to take effect:

$ service nectus-agent restart

Check it’s status to ensure proper operation:

$ service nectus-agent status

If case of successful start you’ll see some output like this:

$ service nectus-agent status

● nectus-agent.service – Nectus Agent

Loaded: loaded (/lib/systemd/system/nectus-agent.service; disabled; vendor preset: enabled)

Active: active (running) since Mon 2019-04-29 17:55:39 UTC; 4s ago

Process: 10247 ExecStart=/usr/bin/nectus-agent $CONFFILE (code=exited, status=0/SUCCESS)

Main PID: 10255 (nectus-agent)

Tasks: 2

Memory: 1.2M

CPU: 14ms

CGroup: /system.slice/nectus-agent.service

└─10255 /usr/bin/nectus-agent /etc/nectus/nectus-agent.conf

Apr 29 17:55:39 ubuntu-16 systemd[1]: Starting Nectus Agent…

Apr 29 17:55:39 ubuntu-16 systemd[1]: nectus-agent.service: PID file /run/nectus/nectus-agent.pid not readable

Apr 29 17:55:39 ubuntu-16 systemd[1]: Started Nectus Agent.

User account

For security reasons it’s not allowed to run Monitoring Agent as root user. If it’s trying to start as ‘root’ then the agent will drop its privileges to the user account set in configuration file

(usually /etc/nectus/nectus-agent.conf) as USER_INSTEAD_OF_ROOT.

Default USER_INSTEAD_OF_ROOT is ‘nectus’.

The agent automatically creates this user account during installation (if this user does not exist already) and provides minimal set of permissions required to run the agent code.

Troubleshooting

Nectus agent runs as a daemon process. You can use the following commands to start, stop restart and check status of the agent:

$ service nectus-agent start

$ service nectus-agent stop

$ service nectus-agent restart

$ service nectus-agent status

If there are any problems with starting the agent first check the agent’s status:

$ service nectus-agent status

Then check the log file (usually /var/log/nectus/nectus-agent.log). If it does not help then you can try to run the agent manually. Usage of the module:

nectus-agent <config-file> [-D]

config-file full path to configuration file

-D debug mode (agent starts not as a daemon but as a usual process)

Locate the executable nectus-agent in /usr/bin and try to run the agent not as a daemon:

$ /usr/bin/nectus-agent /etc/nectus/nectus-agent.conf -D

Check the output and log files at /var/log/nectus.

Upgrade

Upgrade to a new version of the agent is the same as a new installation: download the appropriate package and run either:

> sudo dpkg -i <package-name>.deb

for Debian/Ubuntu systems or

> sudo rpm -U <package-name>.rpm

for RedHat/CentOS.

Uninstallation

To uninstall Nectus agent run the following command:

> sudo dpkg -r nectus-agent

for Debian/Ubuntu systems or

> sudo rpm -e nectus-agent

for RedHat/CentOS.

Appendix 1. Default Configuration File

#########################################
####       GENERAL PARAMETERS        ####
AGENT_IP                0.0.0.0          # IP address of listening socket. 
                                         # Default 0.0.0.0 (listen on all interfaces)
 
AGENT_PORT              5400             # TCP/IP port of the listening socket. Default 5400
 
SSL_CERTIFICATE_DIR     /etc/nectus      # Path to agent’s SSL certificate. 
        # Directory should contain files:
        # dh.pem, certificate.pem and key.pem.
        # To generate those files (self-signed certificate) run commands:
        # openssl dhparam -out dh.pem 1024 
        # openssl req -newkey rsa:2048 -nodes -keyout key.pem -x509 -days 10000 -out certificate.pem
        # Maybe you would need root privileges for this, then run: sudo openssl etc…
 
SERVER_PRESHARED_KEY    <YourSecretKey> # Pre-shared key to encrypt messages between 
                                        # the agent and Nectus monitoring server. 
                                        # Must be the same as set in Nectus GUI.
 
NUM_OF_THREADS          1               # Number of agent’s threads. Valid values are from 1 to 32.
                                        # Default value = 1.
 
USER_INSTEAD_OF_ROOT    nectus          # Drop privileges to a specific user existing in the system.
                                        # Only has effect if the agent is run as root.
                                        # Default value = nectus.
 
#########################################
######### Logging parameters ############
LOG_DIR                         /var/log/nectus  # Path to log files
 
LOG_LEVEL                       info
# Log verbosity level. Possible values are:
#       error – only errors are reported in log
#       info  – default logging level
#       debug – most verbose level including debug information.
 
LOG_FLUSH_PERIOD_SECONDS 30     # Flush log data to disk every X seconds. 
                                # Valid values are from 1 to 900. Default value = 30.
 
LOG_FILE_MAX_SIZE_MBYTES 30     # Maximum size of a single log file. After reaching this size 
                                # next log file is opened. Valid values are from 1 to 1000.
                                # Default value = 30.
 
LOG_FILE_MAX_COUNT      30      # Maximum count of log files stored. After reaching this count
                                # the oldest log files are deleted. 
                                # Valid values are from 1 to 1000. Default value = 30.

ClickHouse DB Installation for Nectus Netflow & Syslog Storage

, ,

ClickHouse Database Installation for Nectus Netflow & Syslog Storage

Requirements: Ubuntu Server 18.04.2 LTS (with SSH access)

NOTE: Although ClickHouse can be installed on several different flavors of Linux, Ubuntu Server 16.04 & 18.04 are the only supported Linux distributions for Nectus at this point.

More information about installation on other OS’s can be found here: https://clickhouse.yandex/docs/en/getting_started/

Step 1

Import the public key:

apt-get update

sudo apt-key adv –keyserver keyserver.ubuntu.com –recv C8F1E19FE0C56BD4

NOTE: It is recommended to import the public key if it’s a fresh Ubuntu install.

Otherwise you may get the following error when adding the repository:GPG error: http://repo.yandex.ru/clickhouse/deb/stable main/ Release: The following signatures couldn’t be verified because the public key is not available: NO_PUBKEY C8F1E19FE0C56BD4

Optional commands to run:

sudo apt-key adv –keyserver keyserver.ubuntu.com –recv E0C56BD4

Step 2

Create Clickhouse repository:

sudo apt-get install dirmngr

sudo apt-add-repository “deb http://repo.yandex.ru/clickhouse/deb/stable/ main/”

NOTE: Please edit the sources.list file if you receive the following error:

“E: Mailformed entry 55 in list file /etc/apt/sources.list”.

Delete the entry XX and save/exit the file. Perform the update (and upgrade if you wish):

sudo apt-get update

Step 3

Get the packages list for the latest updates and the dependencies:

apt-get update && apt-get upgrade

Step 4

Download and Install the ClickHouse packages:

sudo apt-get install -y clickhouse-server clickhouse-server-common clickhouse-client

sudo apt-get install –allow-unauthenticated clickhouse-server-common clickhouse-client clickhouse-server

Step 5

Start the Clickhouse server as a daemon:

sudo service clickhouse-server start

Step 6

Now that we have installed the ClickHouse, it is time to test:

NOTE: TCP ports 8123 & 9000 must be open.

Start the Client:

clickhouse-client

Step 7

Create the Netflow/Syslog database.

In this example we are creating the netflow database named NETFLOW (database name is arbitrary):

clickhouse-client

create database NETFLOW

Step 8

We will now add the Clickhouse internal user “root” with password “nectus” to the Users.xml file located at: /etc/clickhouse-server/users.xml

NOTE: Paste the snippet of the code below starting at line 31 after “<users>” in the users.xml file. You can use vi or nano text editor to edit the file. WinSCP can also be used to accomplish this task. Use the file change owner command if needed “sudo chown -R xxxx users.xml”, where “xxxx” is the user that will take over the ownership of the file.

<root>

<password>nectus</password>

<networks incl=”networks” replace=”replace”>

<ip>::/0</ip>

</networks>

<profile>default</profile>

<quota>default</quota>

</root>

Save the file and exit.

Step 9

Restart the ClickHouse Server:

clickhouse-server restart

Step 10

Now that the ClickHouse Server has been restarted, we can start the ClickHouse client using the internal user that we created in step 8:

clickhouse-client –user default –password nectus

Step 11

We have completed the ClickHouse installation. This last step requires login to Nectus to finish the Netflow/Syslog integration.

Open to “Nectus Settings -> General Settings -> Netflow Integration” page

Enter the required information and click Test DB Connection (Remote Server IP is the IP address of the Ubuntu/ClickHouse server). The result should be “Test DB Connection OK”

Click “Run Integration Scripts” and finally Save.

 

Step 12

Restart Nectus NetFlow and Syslog Services.

Monitoring BGP Sessions with Nectus

,

Monitoring BGP Sessions with Nectus

Nectus 1.2.53 introduces ability to automatically discover and monitor iBGP and eBGP sessions across all router types and manufacturers.

Nectus scheduled network discovery automatically finds all the BGP sessions via SNMP polls and adds it to Nectus database where user can group them by type and enable active monitoring and alerting.

BGP Session Discovery

As part of regular scheduled network discovery Nectus detects all existing BGP Sessions and adds it to one of the existing BGP group which has “Default” flag set to ON.

You can see all the BGP sessions that were discovered in Inventory -> BGP Sessions or in left side panel called “BGP Sessions

 

If you right-click on BGP Session and select “Properties

 

.. you can assign a human friendly name or change a group membership for any of the BGP sessions.

 

You can create multiple BGP session groups and assign discovered BGP sessions to different groups according to its type or functional purpose.

To create a new Group, right-click on “BGP Sessions” line and select “Add New Group” on context menu.

 

Configure New Group name, set “Enable monitoring” to ON if you want all BGP Sessions in this group to be monitored for UP/DOWN Status.

 

If you set “Default” value to ON, this group will become a group where all newly discovered BGP sessions be added automatically. Only one group can have “Default” value set to ON

You can also assign BGP Session to a Group by right-clicking on BGP session and selecting “Move BGP Session to..” option

 

 

By clicking “View Session Info” on BGP Session context menu you can view detailed information about BGP Session

 

 

After you finished creating BGP session groups, assigning BGP sessions to corresponding groups and enabling BGP session groups for Monitoring the final step is to enable BGP Sessions monitoring in corresponding SNMP Device Monitoring Profiles that are applied to the routers that terminate all the BGP sessions.

Find required SNMP Monitoring Profile and enable “BGP Session Status” Monitoring Option

Now are you all set to start monitoring your BGP sessions line a pro.

 

Features coming up in next Nectus release

  1. BGP sessions monitoring with UP/DOWN Alerting
  2. Linux Servers Monitoring (CPU, RAM,  Storage, Up/Down)

UPS Health Status Monitoring with Nectus

,

UPS Status Monitoring with Nectus

Health monitoring of UPS units is one of the most critical tasks of network infrastructure monitoring.

If UPS fails during power outage it results in immediate network downtime resulting in loss of revenue and degrading user experience.

Most of the modern UPS Units perform daily self-tests and results of those self-tests are available via SNMP protocol.

Our goal is to collect those self-test results on regular basis and generate an alert or incident when self-test indicates a UPS failure.

Step 1. Determine which SNMP OID to use

Very first step before you can start monitoring UPS health status is to determine which SNMP OID contains self-test information for your UPS units.

You can open a support case with your UPS vendor and request SNNP OID string for self-test information.

For example, SNMP OID for self-test information from UPS manufactured by APC is .1.3.6.1.4.1.318.1.1.1.7.2.3.0

This OID returns following values:

1 = Ok

2 = Failed

3 = Invalid Test

4 = Test In Progress

 

Step 2. Create Device View that contain all UPS units

Next step is to create a Device View that contain all UPS units that we want to monitor.

To create a Device View go to Inventory -> Views -> SNMP Device Views.

 

Create a new Device Vice called “UPS Units”

 

 

Step 3. Create Custom SNMP Tracker for UPS Self-Test Info

Now we are ready to create a custom SNMP tracker that will be polling our UPS units every 5 min for self-test information.

In Nectus GUI go to Monitoring → SNMP Monitoring Settings → Custom SNMP Trackers

 

Press “Create” button to bring up Custom SNMP Tracker creation interface.

Complete tracker General information tab with correct SNMP OID and select Device View that contains all UPS units that we want to monitor.

 

Select Email and SMS list for Alert Recipients.

In “Alerts” Tab we need to specify which status values will be considered Normal and which values should trigger Alerts.

You can specify multiple values separated by comma.

 

Define an Alert Severity level for Alarm Values as Informational, Warning or Critical.

Define number of consecutive readings for which UPS self-test status must report an Alarm condition before formal Alert is created.

Nectus performs one SNMP poll per 5 min.

If you define value for consecutive readings as 3 it should result in Alert created after 15 minutes of True Alarm conditions.

Press “Save” to complete Custom SNMP tracker creation for UPS Status Monitoring

This completes creation of custom tracker that will be monitoring UPS self-test information on all APC UPS units in your network.

 

Suppression of Frequent Identical Alerts In Nectus

,

Suppression of Frequent Identical Alerts In Nectus

Often when a network device or interface changes its state between Up and Down multiple times in a short time interval, numerous identical alerts will be sent to all the configured alert recipients.

Nectus offers the ability to suppress frequent identical alerts to reduce informational noise level by specifying a suppression timer for up to three hours.

Nectus calculates a CRC hash value from each of the email alert bodies and if hash values of two alerts are matching they are considered to be identical and can be suppressed.

Nectus excludes event timer values and specific metric values from hash calculation logic. You can configure the suppression timer from 0 to 180 minutes where 0 means no suppression.

To configure the suppression timer, go to    Monitoring -> Global Monitoring Settings

 

 

Under Monitoring Settings, select the Alerts tab.

To enable suppression specify a time interval from 1 to 180 minutes.

At the end of the time interval, alert logic will resume as normal.

Setting timer to 0 will disable the suppression.

 

 

Importing WMI Servers from CSV file to Nectus

,

Importing WMI Servers from CSV file to Nectus

To import a list of Windows Servers from CSV file to Nectus prepare a CSV file with only single column containing list of IP addresses of Windows Servers.

No other information is required.

File format example in Excel.

To start import, right-click on WMI Server Group that you want to import servers into and select “Import  WMI Servers from CSV file” option

 

 

Select CSV File that you prepared.

 

 

Press “Start” button

 

Nectus will import each server individually and collect basic Server information via WMI Interface.

 

Configuring WMI Integration in Nectus

, ,

Configuring WMI Integration in Nectus

Nectus uses WMI Interface in several of its modules.

List of modules that depends on WMI Integration:

  1. Windows Server Monitoring Module
  2. IPAM Module

In Windows Server Monitoring module Nectus uses WMI interface to poll Windows Servers for critical health metrics such as CPU and RAM Utilization etc.

In IPAM Module Nectus uses WMI interface to interact with Microsoft DNS and DHCP servers while Importing and creating reservations.

WMI Integration settings must be complete before Nectus can communicate with Windows Servers in those modules.

To complete WMI Integration go to Settings -> General Settings -> WMI Integration

Click on “WMI Integration” link and complete all the required fields for WMI Service account

that will be used by Nectus connect to Windows Servers via WMI Interface.

WMI Service account should not have password expiration policy to prevent Integration from breaking every time password has to be changed.

Use “Test” button to test Service account credential against any of Windows Servers of your choice.

After connection “Test” is passed WMI Integration is complete.