8 hours ago8 hr LibreNMS is what we use to watch switches, routers, firewalls, UPSes and access points across our client sites. When a closet switch drops, we want the alert to reach whoever is on call before the office staff come in and find it. LibreNMS auto-discovers a lot over SNMP, graphs ports and health sensors out of the box, and can alert to chat, ticketing and paging systems. This is how we set up SNMPv3 on a device, add it, write alert rules and wire up transports. I've also included a few transport bugs that cost us alerts before we caught them.What to have readyA working LibreNMS install. We run it in a container, but a native install is the same from here on.Network reachability from the LibreNMS poller to the device on UDP 161Admin access to the deviceWebhook or API details for wherever alerts should go, such as Discord, your ticket system or PagerDutySNMPv3 on the deviceUse SNMPv3 with authentication and privacy (authPriv). SNMPv2c community strings travel in clear text. Here's a generic Cisco IOS example with a read-only view and an ACL limiting who can poll:ip access-list standard SNMP-POLLERS permit 10.10.20.5 ! snmp-server view LIBRENMS-VIEW iso included snmp-server group LIBRENMS-GRP v3 priv read LIBRENMS-VIEW access SNMP-POLLERS snmp-server user librenms LIBRENMS-GRP v3 auth sha AUTH-PASSPHRASE priv aes 128 PRIV-PASSPHRASE snmp-server location Main Office MDF snmp-server contact noc@contoso.comOn IOS, SNMPv3 users don't show up in the running config, so put the credentials in your password manager right away. You can check the user with show snmp user. It's also worth setting a meaningful location string, because LibreNMS uses it for maps and grouping.Prove SNMP works before LibreNMS gets involvedI always test with snmpwalk from the LibreNMS host before adding a device. It tells you whether you have an SNMP problem or a LibreNMS problem:snmpwalk -v3 -l authPriv -u librenms -a SHA -A 'AUTH-PASSPHRASE' -x AES -X 'PRIV-PASSPHRASE' 10.10.20.1 sysDescrIf that returns the device description, add it from the web UI (Devices, Add Device) or the CLI:cd /opt/librenms sudo -u librenms ./lnms device:add --v3 -l authPriv -u librenms \ -a SHA -A 'AUTH-PASSPHRASE' -x AES -X 'PRIV-PASSPHRASE' 10.10.20.1Discovery runs, and the device fills in ports, sensors and hardware details over the next few polling cycles.A short list of alert rulesLibreNMS ships a set of default rules you can import from Alerts, Alert Rules. Start small. An alert stream nobody reads is worse than none. These are the rules we always deploy:Device down, meaning device status is 0 with ICMP and SNMP both failingPort down on uplinks, scoped to ports with a description you tag rather than every access portSensor over limit, for temperature, PSU and UPS battery stateDevice rebooted, meaning uptime below a few minutes, which catches power eventsUse device groups, for example one per client or site, to scope rules and route them to the right transport. Set a sensible delay so a single missed poll doesn't page anyone.Where the alerts goTransports are what deliver alerts. Under Alerts, Alert Transports, add one per destination:For Discord, paste the channel webhook URL. It's good for team visibility.Most ticket systems can create tickets by email, or through an API transport that POSTs JSON to an endpoint.For PagerDuty, use the integration key from a PagerDuty service. We only page for real outages, so nobody gets woken up for a warning.Map rules to transports directly on the rule or through transport groups. Press the Test button on each transport before you rely on it.Proving it end to endRun the validator and fix anything it flags:cd /opt/librenms sudo -u librenms ./validate.phpThe device shows recent poll times and the graphs are filling in.Each transport's Test button delivers a message.Unplug a lab device or shut a test port, and confirm the alert fires, appears in the alert log, and clears when you put it back.The bugs that cost us alertsOn the version we were running, an API transport saved without its options field populated (even as empty) threw a PHP error when an alert fired. It didn't just skip that transport, it stopped the whole alert run. If alerts stop arriving everywhere at once, go looking for one broken transport.A PagerDuty transport created without the region field set caused the same kind of error and killed the run. Open each PagerDuty transport, set the region explicitly, and save it.If you write your own webhook receiver, look at the actual JSON. Fields like alert state arrived as strings in our payloads, and a receiver comparing them to integers silently dropped every alert.There's no automatic retry. If a transport endpoint is down when the alert fires, that notification may simply be lost, so monitor your receivers too.The alert log in the UI shows what fired. When something fired but never arrived, the answer is usually in /opt/librenms/logs/librenms.log.If you add a second poller, add it to the device ACL too. Otherwise it fails with timeouts that look a lot like wrong credentials.After any change to transports, I press Test on every one of them. One broken transport can silence all the others, and you won't find out until a real outage goes unreported.
Create an account or sign in to comment