Unpredictable Juniper SNMPv3 Access Error

Question

We're having issues where SNMPv3 ceases to accept SNMP requests on our Juniper EX series switches. In what seems like random occurrences, Juniper nodes refuse SNMP authentication credentials.

This is what we end up getting:

rj@MHN00525SYS01:~$ snmpwalk -v 3 -a SHA -A "PASS" -l authPriv -x AES -X "PASS" 
  -u "username" mhn00416an01 iso.3.6.1.2.1.1.1.0
snmpwalk: Unknown user name
rj@MHN00525SYS01:~$

Our devices were working, but then just stop. The usernames are correct and loaded into the system.

rj@MHN00416AN01# run show snmp v3 users

Engine ID: 80 00 0a 4c 04 31 33 <...snip...>
    User                            Auth/Priv   Storage      Status
    username                         sha/aes128 nonvolatile  active

After dealing with this for some time, we finally decided to investigate. I'm seeing this entry before the system stops accepting requests:

Nov  6 09:44:32  MHN00416AN01 mgd[7996]: UI_COMMIT: User 'rj' requested 'commit' 
  operation (comment: none)
Nov  6 09:44:35  MHN00416AN01 snmpd[1260]: LIBJSNMP_NS_LOG_ERR: ERR: snmpd: send_trap: USM 
  unknown security name (no such user exists) (plaintext scopedPDU header type 00: s/b 30)
Nov  6 09:44:35  MHN00416AN01 snmpd[1260]: SNMPD_ENGINE_ID_CHANGED: Warning: Local 
  engine ID has changed. Passwords for all the users under [edit snmp v3 usm local-engine] 
  need to be reconfigured.

I have found that simply re-inputting the username/passwords restore access to the device.

rj@MHN00416AN01# set snmp v3 usm local-engine user username 
  authentication-sha authentication-password PASS

{master:0}[edit]
rj@MHN00416AN01# set snmp v3 usm local-engine user username 
  privacy-aes128 privacy-password PASS

After a commit operation, we can access it again.

rj@MHN00525SYS01:~$ snmpwalk -v 3 -a SHA -A "PASS" -l authPriv -x AES -X "PASS" 
  -u "username" mhn00416an01 iso.3.6.1.2.1.1.1.0
iso.3.6.1.2.1.1.1.0 = STRING: "Juniper Networks, Inc. ex3300-48p Ethernet Switch, kernel 
  JUNOS 12.3R3.4, Build date: 2013-06-14 02:21:01 UTC Copyright (c) 1996-2013 Juniper 
  Networks, Inc."
rj@MHN00525SYS01:~$

This issue is happening across a ton of different devices. Models include EX2200-c, EX3300, EX4200 and EX4550 on version 12.3.

Why is this happening and how can I make it stop?

Answer 1

This issue is a result of the engine-id changing. If you run into this syslog message, that is the exact definition of the problem.

Nov  6 09:44:35  MHN00416AN01 snmpd[1260]: SNMPD_ENGINE_ID_CHANGED: Warning: Local 
  engine ID has changed. Passwords for all the users under [edit snmp v3 usm local-engine] 
  need to be reconfigured.

Unfortunately, I have found that this is happening for no apparent reason. The commit performed before this engine-id changed was a simple syslog-host addition that resulted in the engine-id changing. Juniper covers the engine-id option in their TechLibrary, which says the default option is to use the default IP address of the device.

After a lot of beating our heads against the desk, we have found that simply defining one with an explicit setting is the best option, as it remains consistent.

rj@MHN00416AN01# set snmp engine-id local ***.**.244.245

{master:0}[edit]
rj@MHN00416AN01# commit
configuration check succeeds
commit complete

Juniper also has a note for changing the engine-id that needs to be done before it will work. It effectively means once you commit a new engine-id, you must also commit another username/password.

Note: SNMPv3 authentication and encryption keys are generated based on the associated passwords and the engine ID. If you configure or change the engine ID, you must commit the new engine ID before you configure SNMPv3 users. Otherwise the keys generated from the configured passwords are based on the previous engine ID. For the engine ID, we recommend using the MAC address of the management port.

You'll need to commit new users into the configuration after the engine change. This is important because SNMPv3 users won't function if you don't do this despite what is actually said under the show snmp v3 users. So make sure you define new users.

rj@MHN00416AN01# set snmp v3 usm local-engine user username 
  authentication-sha authentication-password PASS

{master:0}[edit]
rj@MHN00416AN01# set snmp v3 usm local-engine user username 
  privacy-aes128 privacy-password PASS

{master:0}[edit]
rj@MHN00416AN01# commit
configuration check succeeds
commit complete

---

Before the change:

rj@MHN00416AN01# run show snmp v3

Local engine ID: 80 00 0a 4c 01 ** ** f4 f5
Engine boots:           1
Engine time:       421043 seconds
Max msg size:       65507 bytes

Engine ID: 80 00 0a 4c 04 31 33 <...snip...>
    User                            Auth/Priv   Storage      Status
    Username                         sha/aes128 nonvolatile  active

After the change:

Local engine ID: 80 00 0a 4c 04 31 33 <...snip...>   <---- previously Engine ID
Engine boots:           1
Engine time:       421110 seconds
Max msg size:       65507 bytes

Engine ID: local   <------ changed
    User                            Auth/Priv   Storage      Status
    Username                         sha/aes128 nonvolatile  active

This provides a more reliable SNMP setup that isn't prone to engine-id changes and eliminating access. Juniper does recommend that you use the use-mac-address option as opposed to the local engine-id-suffix. This option is susceptible to issues in a virtual chassis setup. If a backup routing engine takes over, the SNMP engine-id will change to the me0 interface of the new master.