Today my one of cisco nexus 9396PX switch got rebooted and this is what i found, based on following logs what could be wrong? We don't have Cisco support for this switch, trying to see if i can get help from community otherwise last option is upgrade software.
Software
BIOS: version 07.41
NXOS: version 7.0(3)I4(7)
BIOS compile time: 10/12/2015
NXOS image file is: bootflash:///nxos.7.0.3.I4.7.bin
NXOS compile time: 6/28/2017 14:00:00 [06/28/2017 16:53:29]
Hardware
cisco Nexus9000 C9396PX Chassis
Intel(R) Core(TM) i3- CPU @ 2.50GHz with 16401396 kB of memory.
Processor Board ID SAL2006Y9CQ
reset reason
N9K# show system reset-reason
----- reset reason for module 1 (from Supervisor in slot 1) ---
1) At 310613 usecs after Thu May 9 18:06:25 2019
Reason: Reset Requested due to Fatal Module Error
Service: System manager
Version: 7.0(3)I4(7)
logs-1
N9K# show cores
VDC Module Instance Process-name PID Date(Year-Month-Day Time)
--- ------ -------- --------------- -------- -------------------------
1 1 1 xbar_client 15716 2019-05-09 18:01:23
1 1 1 xbar_client 25312 2019-05-09 18:06:18
logs-2
N9K# show module internal exceptionlog module 1
********* Exception info for module 1 ********
exception information --- exception instance 1 ----
Module Slot Number: 1
Device Id : 134
Device Name : System Manager
Device Errorcode : 0x00000049
Device ID : 00 (0x00)
Device Instance : 00 (0x00)
Dev Type (HW/SW) : 00 (0x00)
ErrNum (devInfo) : 73 (0x49)
System Errorcode : 0x401e008a Service on linecard had a hap-reset
Error Type : FATAL error
PhyPortLayer : 0x0
Port(s) Affected :
Error Description : xbar_client hap reset
DSAP : 0 (0x0)
UUID : 1 (0x1)
Time : Thu May 9 18:06:25 2019
(Ticks: 5CD4B271 jiffies)
exception information --- exception instance 2 ----
Module Slot Number: 1
Device Id : 241
Device Name : BCM5685X
Device Errorcode : 0xcf130200
Device ID : 241 (0xf1)
Device Instance : 48 (0x30)
Dev Type (HW/SW) : 02 (0x02)
ErrNum (devInfo) : 00 (0x00)
System Errorcode : 0x40390047 internal link between forwarding ASICs down
Error Type : Minor error
PhyPortLayer : Ethernet
Port(s) Affected : Ethernet2/1
DSAP : 0 (0x0)
UUID : 0 (0x0)
Time : Thu May 9 17:59:52 2019
(Ticks: 5CD4B0E8 jiffies)
logs-3
N9K# show system internal xbar sw
======= Global Information =========
db_restored = 0
xbm_iam_almost_active = 0
modules_lock_bmap = 0
global_lock = 0
global_lock gwrap = (nil)
chassis type = 34
fabric mode = 1
fabric speed mode = 42g
fabric speed sequence in progress = False
xbar is fully connected
xbar libdrv_xlink_is_t2_speed_40g() is : FALSE
======= Module Information =========
Module in module 1 (present = 1)
rid 0x2000000 type 0 state 0 sub_type 0 node_id 0x102
sw_card_id 0x12a lc_node_addr 0x102 feature_bits 0x0
xlink_index 0x0
locked_gwrap: (nil)
timer: hdl 0x10763d5c rid 0x2000000 ev_id 0xffff timer_id 0x0 tim_type 0x0
Module in module 2 (present = 0)
Module in module 3 (present = 0)
Module in module 4 (present = 0)
logs-4
N9K# show processes log
VDC Process PID Normal-exit Stack Core Log-create-time
--- --------------- ------ ----------- ----- ----- ---------------
1 xbar_client 15716 N N N Thu May 9 18:01:25 2019
1 xbar_client 24518 N N N Thu May 9 18:03:03 2019
1 xbar_client 24933 N N N Thu May 9 18:04:41 2019
1 xbar_client 25312 N N N Thu May 9 18:06:19 2019
logs-5
N9K# show processes log pid 25312
Service: xbar_client
Description: Xbar Client
Executable: /lc/isan/bin/xbar_client
Started at Thu May 9 18:04:41 2019 (663071 us)
Stopped at Thu May 9 18:06:19 2019 (474464 us)
Uptime: 1 minutes 38 seconds
Start type: SRV_OPTION_RESTART_STATEFUL (24)
Death reason: SYSMGR_DEATH_REASON_FAILURE_HEARTBEAT (9)
Last heartbeat 95.72 secs ago
System image name:
System image version: 7.0(3)I4(7)
PID: 25375
Exit code: signal 6 (no core)
Threads: 25312
CWD: /var/sysmgr/work
RLIMIT_AS: 4294967295
Virtual Memory:
CODE 100CB000 - 101B3CA8
DATA 101B4000 - 101B57C8
BRK 115DD000 - 11771000
STACK FF9C7AC0
TOTAL 618916 KB
Memory Map: 100CB000 xbar_clien 101B4000 xbar_clien D30DE000 mts E30DE000 libmtsdlutils.s E30DF000 libmtsdlutils.s E30E5000 libstathash.s E30E7000 libstathash.s E30E8000 libqosmgr.s
E30EE000 libqosmgr.s E30EF000 libfm.s E30F7000 libfm.s E30FA000 liburiparse.s E30FD000 liburiparse.s E317A000 libvdc_capability.s E317D000 libvdc_capability.s E317E000 libvdc_mgr_c
mn.s E3181000 libvdc_mgr_cmn.s E3183000 libz.so.1.2. E3198000 libz.so.1.2. E3199000 libifmgr.s E31FF000 libifmgr.s E3248000 libuspace_utils.s E324A000 libuspace_utils.s E324B000 lib
pcm_sdb.s E3272000 libpcm_sdb.s E3278000 libltlmap.s E3284000 libltlmap.s E3287000 libsdwraphist.s E3293000 libsdwraphist.s E3295000 libcmd.s E32BE000 libcmd.s E32C1000 libdleft.s E
32C5000 libdleft.s E32C7000 liburi_map.s E32C9000 liburi_map.s E32CA000 libvsh.s E32E8000 libvsh.s E32EA000 libipfibutils.s E32EE000 libipfibutils.s EAAB5000 libbios.s EAADC000 libb
ios.s EAADE000 libavl.s EAAE1000 libavl.s EAAE2000 libutils_cli_callback.s EAAE3000 libutils_cli_callback.s EAAE4000 libexec.s EAAE7000 libexec.s EAAE8000 libvdb.s EAAEA000 libvdb.s
EAAF4000 libdll_obj.s EAAF7000 libdll_obj.s EAAF8000 libsysstr.s EAAFA000 libsysstr.s EAAFB000 libsysmgrcmn.s EAB07000 libsysmgrcmn.s EAB08000 libif_index.s EAB47000 libif_index.s
EAB9D000 librt-2.15.s EABA4000 librt-2.15.s EABA5000 librt-2.15.s EABA6000 libbmp.s EAD81000 libbmp.s EAD84000 libeventseq.s EAD98000 libeventseq.s EAD9E000 libsystem_vdc.s EAD9F000
--More--
logs-6
N9K# show system internal xbar event-history lock
1) Event:E_FU_UNLOCK, length:32, at 444695 usecs after Thu May 9 18:09:09 2019
Status: 0x0
Gwrap: 0x108cd04c Cat: 0x0
Opc:MTS_OPC_LC_INSERTED(1081)
Msg id: 0X00006B8E
Lock type: 1
RID Size: 4
Val : 0x 2000000
2) Event:E_FU_LOCK, length:32, at 444635 usecs after Thu May 9 18:09:09 2019
Status: 0x0
Gwrap: 0x108cd04c Cat: 0x0
Opc:MTS_OPC_LC_INSERTED(1081)
Msg id: 0X00006B8E
Lock type: 1
RID Size: 4
Val : 0x 2000000
3) Event:E_FU_UNLOCK, length:32, at 139624 usecs after Thu May 9 18:09:06 2019
Status: 0x0
Gwrap: 0x108cd04c Cat: 0x0
Opc:MTS_OPC_LC_INSERTED(1081)
Msg id: 0X00004EB4
Lock type: 1
RID Size: 4
Val : 0x 2000000
4) Event:E_FU_LOCK, length:32, at 136670 usecs after Thu May 9 18:09:06 2019
Status: 0x0
Gwrap: 0x108cd04c Cat: 0x0
Opc:MTS_OPC_LC_INSERTED(1081)
Msg id: 0X00004EB4
Lock type: 1
RID Size: 4
Val : 0x 2000000
