Category Archives: Linux

Add Nagios plugins to LibreNMS and setup check_http

This is going to explain how to add Nagios plugins to your existing LibreNMS setup. We are going to use the check_http plugin to monitor a website and SSL certificate. The server we are setting this up on is running the following:
Server Version: 16.04.2 LTS (Xenial Xerus)
Apache Version: Apache/2.4.18 (Ubuntu)
MariaDB Version: 15.1 Distrib 10.0.29-MariaDB
PHP Version: 7.0.18

Install Nagios plugins:

root@stupiderror:~# apt install nagios-plugins

Edit config.php and add the following lines to integrate the Nagios plugins with LibreNMS:

root@stupiderror:~# vi /opt/librenms/config.php
$config['show_services']        = 1;
$config['nagios_plugins']       = '/usr/lib/nagios/plugins';

Now in LibreNMS you should see Services on the Navigation bar between Devices and Ports. Under Services if you choose Add Service you should have a list of all of the plugins under the Type drop down menu.

Edit the LibreNMS cron job and make sure it is running check-services.php every 5 minutes. This was already present for me, if it is missing add it.

root@stupiderror:~# vi /etc/cron.d/librenms
*/5  *    * * *   librenms    /opt/librenms/check-services.php >> /dev/null 2>&1

You can add a service from the Services menu or you can go to devices and choose a device. We are going to add the HTTP service to monitor the status code, response time, and SSL certificate expiration. Choose your device and then click on Services which should be in between Inventory and Logs. Click Add Service and choose http for the type. Fill in the description you want. For IP address fill in the website address such as stupiderror.com. If your site redirects to www then fill in something like www.stupiderror.com. For the Parameters we are going to use -S -w 3 -c 5. The -S tells it to connect with ssl so it uses HTTPS instead of HTTP. The -w says to put the service in a Warning state if it takes 3 seconds or longer to respond. The -c option tells it to put the service in a Critical state if it takes 5 seconds or longer to respond.

To monitor the expiration of the certificate click on Add Service again. Choose HTTP for type. For IP address fill in your domain name, something like www.stupiderror.com. For Parameters we can do -C 30,14. This says the service is good when the certificate is valid for more than 30 days. If the certificate is valid for 15 to 30 then days then change the state to Warning and if it is valid for 14 or less days then change the state to Critical. If you click on Details you can see graphs for these metrics.

For more information on check_http check out the man page: https://www.monitoring-plugins.org/doc/man/check_http.html

How to setup LibreNMS on Ubuntu 17.04

This is going to explain how to setup LibreNMS on an Ubuntu 17.04 server. For a production server you should stick with Ubuntu 16.04 as that seems to be what LibreNMS supports. We are going to setup LibreNMS on a server with the following:
Server Version: 17.04 (Zesty Zapus)
Apache Version: Apache/2.4.25 (Ubuntu)
MariaDB Version: 15.1 Distrib 10.1.22-MariaDB
PHP Version: 7.0.18

root@stupiderror:~# apt install apache2 fping git graphviz imagemagick libapache2-mod-fcgid libsnmp-dev mariadb-server mariadb-client mtr-tiny nmap php-cli php-curl php-fpm php-gd php-json php-mcrypt php-mysql php-pear php-snmp python-mysqldb php-net-ipv6 rrdcached rrdtool snmp snmpd whois
root@stupiderror:~# pear install Net_IPV4

LibreNMS needs php-net-ipv4 but this isn’t in the Zesty repositories. Instead we use pear to install it.
Next we will enable the apache modules and then create a database for LibreNMS.


root@stupiderror:~# a2enmod rewrite expires proxy_fcgi setenvif
root@stupiderror:~# a2enconf php7.0-fpm
root@stupiderror:~# phpenmod mcrypt
root@stupiderror:~# systemctl reload apache2.service
root@stupiderror:~# mysql -u root -p
Enter password:  (if you just installed hit enter to continue, no password has been set yet)
MariaDB [(none)]> create database librenms character set utf8 collate utf8_unicode_ci;
MariaDB [(none)]> create user 'librenms'@'localhost' identified by 'uniquepass';
MariaDB [(none)]> grant all privileges on librenms.* to 'librenms'@'localhost';
MariaDB [(none)]> flush privileges;
MariaDB [(none)]> quit

Edit /etc/mysql/mariadb.conf.d/50-server.cnf and add the following lines in bold under the [mysqld] section.


# this is only for the mysqld standalone daemon
[mysqld]

innodb_file_per_table=1
sql-mode=""
innodb_flush_log_at_trx_commit=0

#
# * Basic Settings

Create a librenms user, create folder structure and clone the git repository to install LibreNMS.


root@stupiderror:~# useradd librenms -d /opt/librenms -M -r
root@stupiderror:~# usermod -a -G librenms www-data
root@stupiderror:~# cd /opt
root@stupiderror:/opt# git clone https://github.com/librenms/librenms.git librenms
root@stupiderror:/opt# cd librenms/
root@stupiderror:/opt/librenms# mkdir rrd logs
root@stupiderror:/opt/librenms# chmod 755 rrd

Create an apache site configuration file for Librenms


root@stupiderror:/opt/librenms# vi /etc/apache2/sites-available/librenms.conf

  DocumentRoot /opt/librenms/html
  ServerName librenms.stupiderror.com
  CustomLog /opt/librenms/logs/access_log combined
  ErrorLog /opt/librenms/logs/error_log
  AllowEncodedSlashes NoDecode
  
    Require all granted
    AllowOverride All
    Options FollowSymLinks MultiViews
  

Setup SNMP v3 authentication


root@stupiderror:/opt/librenms# systemctl stop snmpd.service 
root@stupiderror:/opt/librenms# net-snmp-config --create-snmpv3-user -ro -a authPass -x privPass -X AES -A SHA librenms
root@stupiderror:/opt/librenms# curl -o /usr/bin/distro https://raw.githubusercontent.com/librenms/librenms-agent/master/snmp/distro
root@stupiderror:/opt/librenms# chmod +x /usr/bin/distro

Edit /etc/snmp/snmpd.conf and comment out the two lines with the default public string. Then scroll down to find the extend entries and comment out the tests and add the line with distro.


# rocommunity public  default    -V systemonly
# rocommunity6 public  default   -V systemonly

# extend    test1   /bin/echo  Hello, world!
# extend-sh test2   echo Hello, world! ; echo Hi there ; exit 35
#extend-sh test3   /bin/sh /tmp/shtest
extend distro /usr/bin/distro

Restart SNMPD, enable LibreNMS site, disable default site, setup maintenance tasks, and chown the LibreNMS files to librenms user and group.


root@stupiderror:/opt/librenms# systemctl start snmpd.service
root@stupiderror:/opt/librenms# a2ensite librenms.conf
root@stupiderror:/opt/librenms# a2dissite 000-default.conf
root@stupiderror:/opt/librenms# systemctl reload apache2.service
root@stupiderror:/opt/librenms# cp librenms.nonroot.cron /etc/cron.d/librenms
root@stupiderror:/opt/librenms# cp misc/librenms.logrotate /etc/logrotate.d/librenms
root@stupiderror:/opt/librenms# chown -R librenms:librenms /opt/librenms

Now open your browser and go to install.php at your site, in my example I go to http://librenms.stupiderror.com/install.php
All your PHP modules should be installed so you can click Next Stage
Fill in the password for the database you created for LibreNMS and click Next Stage.
Now you should see a status page, mine went to Step #96 and then updated the schema which went to 195. On the last line it says — Done. Click on Goto Add User
Create a user by filling in the info you want to use and click Add User.
Click Generate Config
Copy the config and save it to /opt/librenms/config.php and click Finish Install

Edit /opt/librenms/html/.htaccess and change the last RewriteRule which should like
RewriteRule ^(.*)$ index.php/$1/
change it to:
RewriteRule ^(.*)$ index.php

To enable RRDCached edit /opt/librenms/config.php and uncomment the following line:
$config[‘rrdcached’] = “unix:/var/run/rrdcached.sock”;

Edit /etc/default/rrdcached and modify the following lines like so:


WRITE_TIMEOUT=1800
WRITE_JITTER=1800
BASE_PATH=/opt/librenms/rrd/
SOCKGROUP=librenms
DAEMON_GROUP=librenms
DAEMON_USER=librenms
BASE_OPTIONS="-B -F -R"

root@stupiderror:/opt/librenms# systemctl restart rrdcached.service

Lets finish up with adding the LibreNMS server as a device to be monitored.
Log in to your LibreNMS web portal. Go to Device > Add Device
Fill in the following:
Hostname: 127.0.0.1
SNMP Version: v3
Port: 161

(Previously we used the net-snmp-config to add a user and credentials, I’m going to use the same values as I did above but hopefully you created your own credentials)
Auth Level: authPriv
Auth User Name: librenms
Auth Password: authPass
Auth Algorithm: SHA
Crypto Password: privPass
Crypto Algorithm: AES

Click Add Device

How to setup Lets Encrypt with Apache on Ubuntu 16.10

This is going to explain how to setup Let’s Encrypt for a WordPress site using Apache on Ubuntu. This setup requires you to have root access to the web server.
Server Version: 16.10 (Yakkety Yak)
Apache Version: Apache/2.4.18 (Ubuntu)

First lets generate a Diffie-Helman Parameters file (This will take a while, can easily take 5+ minutes):

stupiderror@localhost:~$ sudo -i
[sudo] password for stupiderror:
stupiderror@localhost:~# openssl dhparam -out /etc/ssl/private/dhparams_4096.pem 4096
Generating DH parameters, 4096 bit long safe prime, generator 2
This is going to take a long time
.....................................................................+..........

Now we will install certbot:

stupiderror@localhost:~# apt update
stupiderror@localhost:~# apt install python-certbot-apache
stupiderror@localhost:~# certbot --apache

Select the domain(s) that you want to enable HTTPS for.
Provide an email address.
Accept the Terms of Service.
Choose Easy or Secure. Easy will let your site be HTTP and HTTPS. Secure will make your site only HTTPS but will redirect anyone trying to use HTTP to HTTPS.

You can use a command like the following and avoid the menu, just substitute in your domain and email:

stupiderror@localhost:~# certbot --apache -d stupiderror.com -d www.stupiderror.com -m support@example.net --agree-tos

The last step is to add a cron entry to auto renew the certificate.

stupiderror@localhost:~# crontab -e
0 1 * * 0    /usr/bin/certbot renew >> /var/log/le-renew.log

Reset WordPress Password

This is going to explain how to reset a WordPress password utilizing MySQL.
This was performed on a server running the following:
Server Version: Ubuntu 16.04.1 LTS (Xenial Xerus)
Wordpress Version: 4.6.1
MySQL Version: 5.7.15
PHP Version: 7.0.8

First we will login to MySQL. If you don’t know your MySQL username, database, and/or password you can find that in wp-config.php at the root of your WordPress site.

/** The name of the database for WordPress */
define('DB_NAME', 'wordpress');

/** MySQL database username */
define('DB_USER', 'stupiderror');

/** MySQL database password */
define('DB_PASSWORD', 'WPpassword');

/** MySQL hostname */
define('DB_HOST', 'localhost');

Use this information to login to MySQL:

stupiderror@localhost:~/html$ mysql -u stupiderror -h localhost -p
Enter password:
mysql> use wordpress;
Database changed
mysql> show tables;
+-----------------------+
| Tables_in_wordpress   |
+-----------------------+
| wp_commentmeta        |
| ...                   |
| wp_users              |
| wp_wpeditor_settings  |
+-----------------------+
17 rows in set (0.00 sec)
mysql>

Most of the tables were truncated from the results. We are looking for a table ending in _users, in this case wp_users. Now lets find the user we want to reset the password for.

mysql> select id, user_login,user_pass,user_email from wp_users;
+----+------------+------------------------------------+-------------------+
| id | user_login | user_pass                          | user_email        |
+----+------------+------------------------------------+-------------------+
|  1 | Admin      | $P$B3pqWu6gdqNO/kzDB4ZtnDPntHIq8f. | adm@localhost.com |
+----+------------+------------------------------------+-------------------+
1 row in set (0.00 sec)
mysql>

If you have a modern version of PHP and WordPress your password hash should start with $P$
This means WordPress is using PHPass. We will use a small script to generate a new hash.
On the 6th line replace newPassword with the new password you want to use. Save this file to the root of your WordPress site.

<?php
  require( dirname(__FILE__) . '/wp-load.php' );
  require_once ABSPATH . WPINC . '/class-phpass.php';
  $hasher = new PasswordHash(8,TRUE);

  $hash = $hasher->HashPassword('newPassword');

  echo "Hash: $hash\r\n"
?>

Now generate a hash and then update the database with the new hash:

stupiderror@localhost:~/html$ php generateHash.php
Hash: $P$B3pqWu6gdqNO/kzDB4ZtnDPntHIq8f.

mysql> update wp_users set user_pass="$P$B3pqWu6gdqNO/kzDB4ZtnDPntHIq8f." where ID = 1;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0
mysql>

Now you should be able to login to your WordPress site with the new password.

Installing Google Authenticator for SSH

This entry shows how to install Google Authenticator and set it up to provide ssh with two-factor authentication. This was done on Ubuntu Server 14.10. This is software that runs on your server and mobile device. It does not require a Gmail account or any third party service.

Install On Server
Install libpam-google-authenticator package:

stupiderror@localhost:~$ sudo -i
[sudo] password for stupiderror:
root@localhost:~# apt-get install libpam-google-authenticator
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
  libqrencode3
The following NEW packages will be installed:
  libpam-google-authenticator libqrencode3
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 59.3 kB of archives.
After this operation, 206 kB of additional disk space will be used.
Do you want to continue? [Y/n] y

Setup User
Now we will setup Google Authenticator for user stupiderror:

root@localhost:~# exit
logout
stupiderror@localhost:~$ google-authenticator

Do you want authentication tokens to be time-based (y/n) y
Your new secret key is: BHBWGPOSWB6K19QY
Your verification code is 674538
Your emergency scratch codes are:
  16572244
  40474860
  4213D034
  55121160
  04767458

Do you want me to update your "/home/stupiderror/.google_authenticator" file (y/n) y

Do you want to disallow multiple uses of the same authentication
token? This restricts you to one login about every 30s, but it increases
your chances to notice or even prevent man-in-the-middle attacks (y/n) y

By default, tokens are good for 30 seconds and in order to compensate for
possible time-skew between the client and the server, we allow an extra
token before and after the current time. If you experience problems with poor
time synchronization, you can increase the window from its default
size of 1:30min to about 4min. Do you want to do so (y/n) n

If the computer that you are logging into isn't hardened against brute-force
login attempts, you can enable rate-limiting for the authentication module.
By default, this limits attackers to no more than 3 login attempts every 30s.
Do you want to enable rate-limiting (y/n) y

We choose to make the one time tokens time based. By default they are displayed for 30 seconds and have a 60 second grace period. Next it shows a QR code which was omitted above. This assists with getting it setup on your phone which we will get to next. If you don’t want to use the QR code you can enter the secret key on your phone instead. It then provides 5 emergency scratch codes. Write these down and they can be used when you need to login and your phone isn’t available. They are not time sensitive and each code can only be used once.

We choose yes to disallow the authentication token from being used more than once. If you want to log in to the server and you look to open two ssh sessions this means you can use the verification code displayed on your mobile device and then must wait for the next verification code to be displayed to complete the second login. If an attacker is able to sniff your keystrokes this means they cannot use the verification code you just used to log in.

We also enable rate limiting to slow down brute force attacks. Sshguard is installed and works fine with this enabled.

Setup Phone
On your phone search for Google Authenticator App in the app store and install it. It is made by Google, Inc. Open Google Authenticator on your phone and press the + in the top right to add an account. Choose Scan barcode or Manual entry. If you choose Scan barcode then point the camera on your phone at the QR code that was displayed when you ran google-authenticator on the server. Once it sees the code it adds it and names it after the user and hostname of the server google-authenticator was run as. In this example it named it: stupiderror@localhost. If you would like to rename the entry then click the pencil icon in the top right on your mobile device. Then the name will have a pencil icon next to it and you can click the name and edit it.

If you choose manual entry then fill in the information requested. Account is just a friendly name for you to identify the server. You can set this up on multiple servers and/or multiple users and add them all. The account name allows you to differentiate them. For key you need to enter the secret key that was displayed after the QR code when you ran google-authenticator. In the above example the key is BHBWGPOSWB6K19QY. Do not use this key, you need to use the one that was generated for your account. Then enable or disable the time restriction, we are leaving it enabled.

Enable Google Authenticator On Server
The time and date settings are important. If the time and date on the server is too far off from the phone the time based verification codes will fail. Make sure the time and date are set correctly. You may want to make a crontab entry to sync it daily:

0 0 * * *      /usr/sbin/ntpdate -s pool.ntp.org


We need root to edit these files:

stupiderror@localhost:~$ sudo -i
[sudo] password for stupiderror:
root@localhost:~# vi /etc/ssh/sshd_config


Find this line:

ChallengeResponseAuthentication no


and change the no to yes

ChallengeResponseAuthentication yes


Edit /etc/pam.d/sshd

root@localhost:~# vi /etc/pam.d/sshd


Add the following line and it should be before “@include common-auth”:

auth	required			pam_google_authenticator.so


To enable these changes restart SSHD:

root@localhost:~# service ssh restart

Test It
I would not disconnect your current session because if something goes wrong you will need to login as a different user or fix it from the console.

Using keyboard-interactive authentication.
Verification code:
Using keyboard-interactive authentication.
Password:
Last login: example.net
stupiderror@localhost:~$

If it isn't working properly the best place to look for error messages is probably the auth.log

root@localhost:~# tail -f /var/log/auth.log