Skip to content
View in the app

A better way to browse. Learn more.

Learn more

The LibreNMS setup we use at client sites, plus the transport bugs that quietly ate our alerts

Featured Replies

LibreNMS is what we use to watch switches, routers, firewalls, UPSes and access points across our client sites. When a closet switch drops, we want the alert to reach whoever is on call before the office staff come in and find it. LibreNMS auto-discovers a lot over SNMP, graphs ports and health sensors out of the box, and can alert to chat, ticketing and paging systems. This is how we set up SNMPv3 on a device, add it, write alert rules and wire up transports. I've also included a few transport bugs that cost us alerts before we caught them.

What to have ready

  • A working LibreNMS install. We run it in a container, but a native install is the same from here on.
  • Network reachability from the LibreNMS poller to the device on UDP 161
  • Admin access to the device
  • Webhook or API details for wherever alerts should go, such as Discord, your ticket system or PagerDuty

SNMPv3 on the device

Use SNMPv3 with authentication and privacy (authPriv). SNMPv2c community strings travel in clear text. Here's a generic Cisco IOS example with a read-only view and an ACL limiting who can poll:

ip access-list standard SNMP-POLLERS
 permit 10.10.20.5
!
snmp-server view LIBRENMS-VIEW iso included
snmp-server group LIBRENMS-GRP v3 priv read LIBRENMS-VIEW access SNMP-POLLERS
snmp-server user librenms LIBRENMS-GRP v3 auth sha AUTH-PASSPHRASE priv aes 128 PRIV-PASSPHRASE
snmp-server location Main Office MDF
snmp-server contact noc@contoso.com

On IOS, SNMPv3 users don't show up in the running config, so put the credentials in your password manager right away. You can check the user with show snmp user. It's also worth setting a meaningful location string, because LibreNMS uses it for maps and grouping.

Prove SNMP works before LibreNMS gets involved

I always test with snmpwalk from the LibreNMS host before adding a device. It tells you whether you have an SNMP problem or a LibreNMS problem:

snmpwalk -v3 -l authPriv -u librenms -a SHA -A 'AUTH-PASSPHRASE' -x AES -X 'PRIV-PASSPHRASE' 10.10.20.1 sysDescr

If that returns the device description, add it from the web UI (Devices, Add Device) or the CLI:

cd /opt/librenms
sudo -u librenms ./lnms device:add --v3 -l authPriv -u librenms \
  -a SHA -A 'AUTH-PASSPHRASE' -x AES -X 'PRIV-PASSPHRASE' 10.10.20.1

Discovery runs, and the device fills in ports, sensors and hardware details over the next few polling cycles.

A short list of alert rules

LibreNMS ships a set of default rules you can import from Alerts, Alert Rules. Start small. An alert stream nobody reads is worse than none. These are the rules we always deploy:

  • Device down, meaning device status is 0 with ICMP and SNMP both failing
  • Port down on uplinks, scoped to ports with a description you tag rather than every access port
  • Sensor over limit, for temperature, PSU and UPS battery state
  • Device rebooted, meaning uptime below a few minutes, which catches power events

Use device groups, for example one per client or site, to scope rules and route them to the right transport. Set a sensible delay so a single missed poll doesn't page anyone.

Where the alerts go

Transports are what deliver alerts. Under Alerts, Alert Transports, add one per destination:

  • For Discord, paste the channel webhook URL. It's good for team visibility.
  • Most ticket systems can create tickets by email, or through an API transport that POSTs JSON to an endpoint.
  • For PagerDuty, use the integration key from a PagerDuty service. We only page for real outages, so nobody gets woken up for a warning.

Map rules to transports directly on the rule or through transport groups. Press the Test button on each transport before you rely on it.

Proving it end to end

Run the validator and fix anything it flags:

cd /opt/librenms
sudo -u librenms ./validate.php
  • The device shows recent poll times and the graphs are filling in.
  • Each transport's Test button delivers a message.
  • Unplug a lab device or shut a test port, and confirm the alert fires, appears in the alert log, and clears when you put it back.

The bugs that cost us alerts

  • On the version we were running, an API transport saved without its options field populated (even as empty) threw a PHP error when an alert fired. It didn't just skip that transport, it stopped the whole alert run. If alerts stop arriving everywhere at once, go looking for one broken transport.
  • A PagerDuty transport created without the region field set caused the same kind of error and killed the run. Open each PagerDuty transport, set the region explicitly, and save it.
  • If you write your own webhook receiver, look at the actual JSON. Fields like alert state arrived as strings in our payloads, and a receiver comparing them to integers silently dropped every alert.
  • There's no automatic retry. If a transport endpoint is down when the alert fires, that notification may simply be lost, so monitor your receivers too.
  • The alert log in the UI shows what fired. When something fired but never arrived, the answer is usually in /opt/librenms/logs/librenms.log.
  • If you add a second poller, add it to the device ACL too. Otherwise it fails with timeouts that look a lot like wrong credentials.

After any change to transports, I press Test on every one of them. One broken transport can silence all the others, and you won't find out until a real outage goes unreported.

Create an account or sign in to comment

Navigation

Search

Search

Configure browser push notifications

Chrome (Android)
  1. Tap the lock icon next to the address bar.
  2. Tap Permissions → Notifications.
  3. Adjust your preference.
Chrome (Desktop)
  1. Click the padlock icon in the address bar.
  2. Select Site settings.
  3. Find Notifications and adjust your preference.