UPDATE ......From Tuesday 8 April 2025 we have changed the way that Single Sign-on works on this wiki. Please see here for more information:
Update
The configuration files are stored in GitLab at the location: https://git.geant.net/gitlab/puppet-apps/dashboard/tree/test. You may have to ask DevOps to grant you access to the dashboard project
Navigate to the https://git.geant.net/gitlab/puppet-apps/dashboard/blob/test/files/etc/dashboard/core/routers_community.conf file and add the router name, SNMP Community String and IP Address in the following format:
mx1.ath2.gr.geant.net=0pBiFbD,62.40.96.39
Commit the change to the GITLab repository and raise a support ticket for the DevOps team to push the change to Test, UAT and Production instances of the current Dashboard.
The physical file on the server is: /usr/lib64/nagios/plugins/dbupdates/conf/routers_community.conf (Should you wish to be naughty and update the file directly on the server!!!)
*** Because the Operations team has commissioned a new router, we want to Dashboard to be able to raise critical alarms should a fault arise. In order for this to happen, we need to run some of the PHP and BASH Shell scripts detailed in the CRONTAB.
Log into prod-newdboard01.geant.net using your dante.surname credentials.
Sudo to root using the command: sudo su -
*** If you can not run the above command, ask the DevOps team to add you to the sudoers file.
Check the CRON jobs that run as the user nagios: crontab -u nagios -l
###############################################################################################################################
#
# This file lists the cron jobs that should be run as part of Dashboard application.
# Please note that the location of scripts may need to change depending on the location where Dashboard (and its scripts) have been installed.
#
###############################################################################################################################
# To be run on: BOTH Primary and Backup Dashboard server
# Process SNMP traps from /var/log/snmptraps/new folder and prepare them for the alarm correlator
* * * * * cd /usr/lib64/nagios/plugins && ./run_once.sh /usr/lib64/nagios/plugins/process-multiple-traps.sh /var/log/snmptraps /usr/lib64/nagios/plugins/process_traps.pl
# Backup and delete SNMP traps at 01:30 on 1st of every month
#30 01 1 * * cd /usr/lib64/nagios/plugins/dbupdates/scripts && ./backupAndDeleteTraps.sh 2>&1
# To be run on: ONLY Primary Dashboard server
# routers table update: updates table with new routers, if any - runs 10 mins past midnight every night
10 0 * * * cd /usr/lib64/nagios/plugins/dbupdates/scripts/ && ./update_routers_table.php > /var/log/nms-dashboard/router_details.log 2>&1
# To be run on: ONLY Primary Dashboard server
# router_details table update: updates table with interfaces for routers - runs 15 mins past the hour, every 4 hours
15 0-23/4 * * * cd /usr/lib64/nagios/plugins/dbupdates/scripts/ && ./populate_router_details_table.php >> /var/log/nms-dashboard/router_details.log 2>&1
# To be run on: ONLY Primary Dashboard server
# get BGP peers that peer with GEANT routers (Dashboard-76)
15 13 * * * cd /usr/lib64/nagios/plugins/dbupdates/scripts/ && ./get-bgp-v4-peers-on-router.sh > /var/log/nms-dashboard/ix_public_peers.log 2>&1
30 13 * * * cd /usr/lib64/nagios/plugins/dbupdates/scripts/ && ./get-bgp-v6-peers-on-router.sh >> /var/log/nms-dashboard/ix_public_peers.log 2>&1
15 14 * * * cd /usr/lib64/nagios/plugins/dbupdates/scripts/ && ./populate_router_ix_public_peer_table.php >> /var/log/nms-dashboard/ix_public_peers.log 2>&1
# To be run on: ONLY Primary Dashboard server
# get VPN RR BGP peers from GEANT routers for MD VPN monitoring (Dashboard-314)
00 19 * * * cd /usr/lib64/nagios/plugins/dbupdates/scripts/ && ./get-bgp-v4-vpnRRPeers-on-router.sh > /var/log/nms-dashboard/vpnrr_peers.log 2>&1
35 19 * * * cd /usr/lib64/nagios/plugins/dbupdates/scripts/ && ./populate_router_vpnrr_peer_table.php >> /var/log/nms-dashboard/vpnrr_peers.log 2>&1
We are now going to manually run the commands highlighted in bold text:
*** Some of the scripts use a configuration file called /usr/lib64/nagios/plugins/dbupdates/conf/config.inc
cd /usr/lib64/nagios/plugins/dbupdates/scripts/
./update_routers_table.php - This inserts records in the alarms.routers table
./populate_router_details_table.php - This inserts records in the alarms.router_details table (This can take some time to complete)
./get-bgp-v4-peers-on-router.sh - It trys to PING each router listed in the /usr/lib64/nagios/plugins/dbupdates/conf/routers_community.conf file, if it gets a response, it then connects to each router using ssh and public keys (/usr/lib64/nagios/plugins/dbupdates/.sshkeys/monitor_dsa and /usr/lib64/nagios/plugins/dbupdates/.sshkeys/monitor_known_hosts). It generates a file called /var/lib/nms-dashboard/v4-<Router_Name>.txt (This can take about 10 minutes to complete)
./get-bgp-v6-peers-on-router.sh - It trys to PING each router listed in the /usr/lib64/nagios/plugins/dbupdates/conf/routers_community.conf file, if it gets a response, it then connects to each router using ssh and public keys (/usr/lib64/nagios/plugins/dbupdates/.sshkeys/monitor_dsa and /usr/lib64/nagios/plugins/dbupdates/.sshkeys/monitor_known_hosts). It generates a file called /var/lib/nms-dashboard/v6-<Router_Name>.txt (This can take about 10 minutes to complete)
./populate_router_ix_public_peer_table.php - This parses the .txt files generated by get-bgp-v4-peers-on-router.sh and get-bgp-v6-peers-on-router.sh, located in the /var/lib/nms-dashboard directory. For each .txt file found, it updates/deletes records from the alarms.router_ix_public_peers table
*** WARNING - The scripts run the equivalent ssh command:
ssh -n -i "/usr/lib64/nagios/plugins/dbupdates/.sshkeys/monitor_dsa" -oPasswordAuthentication=no -oStrictHostKeyChecking=no -oPreferredAuthentications=publickey -oConnectionAttempts=1 -oConnectTimeout=10 -oUserKnownHostsFile=/usr/lib64/nagios/plugins/dbupdates/.sshkeys/monitor_known_hosts Monit0r@mx1.dub.ie.geant.net
The scripts may print out an error/warning such as:
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: POSSIBLE DNS SPOOFING DETECTED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
The RSA host key for mx1.dub.ie.geant.net has changed,
and the key for the corresponding IP address 62.40.96.3
has a different value. This could either mean that
DNS SPOOFING is happening or the IP address for the host
and its host key have changed at the same time.
Offending key for IP in /usr/lib64/nagios/plugins/dbupdates/.sshkeys/monitor_known_hosts:135
To fix the above, we need to delete the line that references the router within the /usr/lib64/nagios/plugins/dbupdates/.sshkeys/monitor_known_hosts, to do this easily do the following:
Change to root by running: sudo su -
cd /usr/lib64/nagios/plugins/dbupdates/.sshkeys/
chmod 666 monitor_known_hosts
Now edit the file and remove all references to the router (both IPv4 and IPv6 references), this could be by router name and by ip address
chmod 600 monitor_known_hosts
Now rerun the command: ssh -n -i "/usr/lib64/nagios/plugins/dbupdates/.sshkeys/monitor_dsa" -oPasswordAuthentication=no -oStrictHostKeyChecking=no -oPreferredAuthentications=publickey -oConnectionAttempts=1 -oConnectTimeout=10 -oUserKnownHostsFile=/usr/lib64/nagios/plugins/dbupdates/.sshkeys/monitor_known_hosts Monit0r@mx1.dub.ie.geant.net
You should see the following:
Pseudo-terminal will not be allocated because stdin is not a terminal.
Warning: Permanently added the RSA host key for IP address '62.40.96.3' to the list of known hosts.
----------------------------------------------------------------
This is mx1.dub.ie.geant.net, a GEANT router in Dublin, Ireland
Warning: Unauthorized access to this equipment is strictly forbidden and will lead to prosecution
-------------------------------------------------------------
{master}
Monit0r@mx1.dub.ie.re0>
Once you have fixed the monitor_known_hosts file, you we need to rerun the get-bgp-v4-peers-on-router.sh, get-bgp-v6-peers-on-router.sh and populate_router_ix_public_peer_table.php scripts.
The router's details also need to be added manually to the juniper_servers table in the alarms database.
- Log into the alarms database using the newdashboardBiz user
- execute the follow statement INSERT INTO juniper_servers VALUES ('<ip_address>', '<router project>');