I've been seeing an ongoing problem with Juniper EX4300s and SRX 345 and 1500s. The problem that we continually see is that our monitoring servers stop receiving traps from these devices. I've had several tickets open with Juniper on this and the problem has yet to be solved. I'm curious if anyone else has seen anything similar.
So, what we notice is that periodically the devices seem to randomly stop sending traps. To prove this, my team has ran a tcpdump on the Junos Space server and then spoofed traps from the devices. The device will say that it generates a trap, but the tcpdump does not ever see the trap come in. Additionally, I've put a network tap in line between the fxp0 port of our SRXs and the access port on the switch. When spoofing a trap, I see no snmptraps coming out of the fxp0 port with a packet capture running.
The frustrating thing is that we cannot replicate the issue. Whatever we do, we cannot get snmp to break when it is working. No combination of rebooting the device or server replicates the issue. The other frustrating part is that we can easily fix it. I've found that simply issuing the "restart snmp" command corrects the issue instantly. As soon as we restart it, the tcpdump and wireshark will show the spoofed traps. I'm to the point where I may just have to schedule an snmp restart job once a week during a maintenance window if we can't get this figured out.
Any ideas?
user@SRX1500-A1> show configuration snmp | display set relative
set v3 usm local-engine user SNMPv3Guest authentication-sha authentication-key "$9$IgtcevNdb2oJx7b2aJHiHmLxk."
set v3 usm local-engine user SNMPv3Guest privacy-aes128 privacy-key "$9$MnhXdbaZUH.P4oUHmP3nM8jiH5QFn/AtO/9"
set v3 usm local-engine user SNMPv3Admin authentication-sha authentication-key "$9$R6jSKM-VwgaZ7NwgJZkq39puEhevW8NdgoJHkPX7qm"
set v3 usm local-engine user SNMPv3Admin privacy-aes128 privacy-key "$9$7gV24jHqf1hu0"
set v3 vacm security-to-group security-model usm security-name SNMPv3Admin group ADMIN
set v3 vacm security-to-group security-model usm security-name SNMPv3Guest group GUEST
set v3 vacm access group ADMIN default-context-prefix security-model any security-level privacy read-view ADMIN-VIEW
set v3 vacm access group ADMIN default-context-prefix security-model any security-level privacy write-view ADMIN-VIEW
set v3 vacm access group ADMIN default-context-prefix security-model any security-level privacy notify-view ADMIN-VIEW
set v3 vacm access group ADMIN context-prefix default-context-prefix security-model any security-level privacy read-view ADMIN-VIEW
set v3 vacm access group ADMIN context-prefix default-context-prefix security-model any security-level privacy write-view ADMIN-VIEW
set v3 vacm access group ADMIN context-prefix default-context-prefix security-model any security-level privacy notify-view ADMIN-VIEW
set v3 vacm access group GUEST default-context-prefix security-model any security-level privacy read-view GUEST-VIEW
set v3 vacm access group GUEST context-prefix default-context-prefix security-model any security-level privacy read-view GUEST-VIEW
set v3 target-address SVR-WUG1 address 2001:db8:33:38c6::1:200
set v3 target-address SVR-WUG1 port 162
set v3 target-address SVR-WUG1 tag-list MGMT
set v3 target-address SVR-WUG1 routing-instance mgmt_junos
set v3 target-address SVR-WUG1 target-parameters ADMIN
set v3 target-address SVR-JSPACE1 address 2001:db8:33:38c4::1:240
set v3 target-address SVR-JSPACE1 port 162
set v3 target-address SVR-JSPACE1 tag-list MGMT
set v3 target-address SVR-JSPACE1 routing-instance mgmt_junos
set v3 target-address SVR-JSPACE1 target-parameters ADMIN
set v3 target-address TA_SPACE address 2001:db8:33:38c4:0:0:1:241
set v3 target-address TA_SPACE tag-list TAG_SPACE
set v3 target-address TA_SPACE target-parameters TP_SPACE
set v3 target-parameters ADMIN parameters message-processing-model v3
set v3 target-parameters ADMIN parameters security-model usm
set v3 target-parameters ADMIN parameters security-level privacy
set v3 target-parameters ADMIN parameters security-name SNMPv3Admin
set v3 target-parameters ADMIN notify-filter ADMIN-TRAPS
set v3 target-parameters GUEST parameters message-processing-model v3
set v3 target-parameters GUEST parameters security-model usm
set v3 target-parameters GUEST parameters security-level privacy
set v3 target-parameters GUEST parameters security-name SNMPv3Guest
set v3 target-parameters TP_SPACE parameters message-processing-model v3
set v3 target-parameters TP_SPACE parameters security-model usm
set v3 target-parameters TP_SPACE parameters security-level privacy
set v3 target-parameters TP_SPACE parameters security-name SNMPv3Admin
set v3 target-parameters TP_SPACE notify-filter SPACE_TRAP_FILTER
set v3 notify ADMIN type trap
set v3 notify ADMIN tag MGMT
set v3 notify GUEST type trap
set v3 notify GUEST tag MGMT
set v3 notify SPACE_TRAPS type trap
set v3 notify SPACE_TRAPS tag TAG_SPACE
set v3 notify-filter ADMIN-TRAPS oid .1 include
set v3 notify-filter SPACE_TRAP_FILTER oid .1 include
set engine-id use-mac-address
set view GUEST-VIEW oid .1 include
set view ADMIN-VIEW oid .1 include
set client-list SNMP_CLIENTS 2001:db8:33:38c4::1:240/128
set client-list SNMP_CLIENTS 2001:db8:33:38c4::1:241/128
set client-list SNMP_CLIENTS 2001:db8:33:38c6::1:200/128
set trap-options routing-instance mgmt_junos source-address 2001:db8:44:3611::4:200
request snmp spoof-trap), deliberately cause a situation triggering a specific trap (run invalid command, ...), or something similar via a scheduler (either on the EX itself or from external by SSH, SNMP, ...). Then monitor the trap receiver if that specific trap was received in time. If more than 1x is overdue, restart the SNMP service.