View Source

Problem: New alarms are not appearing in the gui

Possible Cause: Traps not being processed by RabbitMQ

Analysis

Open one or more of the following RabbitMQ management consoles. (Credentials are in the "GÉANT Dashboard v3" LastPass folder)
Scroll down to the "Nodes" section
There should be 3 rows in the table and all status icons should be green. The expected node hostnames are:
- prod-noc-alarms01.geant.org
- prod-noc-alarms02.geant.org
- prod-dashboard-storage01.geant.org

Solution

If one of the 3 nodes is failing or missing from the list, log into the failing server and restart the RabbitMQ service:
- ```
sudo systemctl restart rabbitmq-server
```
After a minute or two the management consoles should show the cluster is restored.

Possible Cause: Collectors have stopped working

Analysis

Open https://net-alarms-monitoring.geant.org/d/hESYQotZz/correlation-services?orgId=1
Scroll down to the "Collectors" panel
Check that the graph shows a nonzero rate of traps being processes

Solution

On each of the following servers:
- net-alarms01.geant.org
- net-alarms02.geant.org
Log in and execute the following command:
- ```
sudo systemctl restart trap_collector
```

Possible Cause: Correlators have stopped working

Analysis

Open https://net-alarms-monitoring.geant.org/d/hESYQotZz/correlation-services?orgId=1
Scroll down to the "Collectors" panel
Check that the graph shows one of the collector processes processing a non-zero rate of traps.
- note that it is normal for only one of the collectors to be processing traps, the other line should remain at zero.

Solution

On each of the following servers:
- net-alarms01.geant.org
- net-alarms02.geant.org
Log in and execute the following command:
- ```
sudo systemctl restart trap_correlator
```

Related articles

Related articles appear here based on the labels you select. Click to edit the macro and add or change labels.

Related issues