4

Today my one of cisco nexus 9396PX switch got rebooted and this is what i found, based on following logs what could be wrong? We don't have Cisco support for this switch, trying to see if i can get help from community otherwise last option is upgrade software.

Software
  BIOS: version 07.41
  NXOS: version 7.0(3)I4(7)
  BIOS compile time:  10/12/2015
  NXOS image file is: bootflash:///nxos.7.0.3.I4.7.bin
  NXOS compile time:  6/28/2017 14:00:00 [06/28/2017 16:53:29]


Hardware
  cisco Nexus9000 C9396PX Chassis
  Intel(R) Core(TM) i3- CPU @ 2.50GHz with 16401396 kB of memory.
  Processor Board ID SAL2006Y9CQ

reset reason

N9K# show system reset-reason
----- reset reason for module 1 (from Supervisor in slot 1) ---
1) At 310613 usecs after Thu May  9 18:06:25 2019
    Reason: Reset Requested due to Fatal Module Error
    Service: System manager
    Version: 7.0(3)I4(7)

logs-1

  N9K# show cores
    VDC  Module  Instance  Process-name     PID       Date(Year-Month-Day Time)
    ---  ------  --------  ---------------  --------  -------------------------
    1    1       1         xbar_client      15716     2019-05-09 18:01:23
    1    1       1         xbar_client      25312     2019-05-09 18:06:18

logs-2

 N9K# show module internal exceptionlog module 1
    ********* Exception info for module 1 ********

    exception information --- exception instance 1 ----
    Module Slot Number: 1
    Device Id         : 134
    Device Name       : System Manager
    Device Errorcode  : 0x00000049
    Device ID         : 00 (0x00)
    Device Instance   : 00 (0x00)
    Dev Type (HW/SW)  : 00 (0x00)
    ErrNum (devInfo)  : 73 (0x49)
    System Errorcode  : 0x401e008a Service on linecard had a hap-reset
    Error Type        : FATAL error
    PhyPortLayer      : 0x0
    Port(s) Affected  :
    Error Description : xbar_client hap reset
    DSAP              : 0 (0x0)
    UUID              : 1 (0x1)
    Time              : Thu May  9 18:06:25 2019
                        (Ticks: 5CD4B271 jiffies)

    exception information --- exception instance 2 ----
    Module Slot Number: 1
    Device Id         : 241
    Device Name       : BCM5685X
    Device Errorcode  : 0xcf130200
    Device ID         : 241 (0xf1)
    Device Instance   : 48 (0x30)
    Dev Type (HW/SW)  : 02 (0x02)
    ErrNum (devInfo)  : 00 (0x00)
    System Errorcode  : 0x40390047 internal link between forwarding ASICs down
    Error Type        : Minor error
    PhyPortLayer      : Ethernet
    Port(s) Affected  : Ethernet2/1
    DSAP              : 0 (0x0)
    UUID              : 0 (0x0)
    Time              : Thu May  9 17:59:52 2019
                        (Ticks: 5CD4B0E8 jiffies)

logs-3

N9K# show system internal xbar sw
======= Global Information =========
         db_restored = 0
         xbm_iam_almost_active = 0
         modules_lock_bmap = 0
         global_lock = 0
         global_lock gwrap = (nil)
         chassis type = 34
         fabric mode  = 1
         fabric speed mode  = 42g
         fabric speed sequence in progress  = False
         xbar is fully connected
         xbar libdrv_xlink_is_t2_speed_40g() is : FALSE
======= Module Information =========
Module in module 1 (present = 1)

rid 0x2000000 type 0 state 0 sub_type 0 node_id 0x102
sw_card_id 0x12a lc_node_addr 0x102 feature_bits 0x0
xlink_index 0x0
locked_gwrap: (nil)
timer: hdl 0x10763d5c rid 0x2000000 ev_id 0xffff timer_id 0x0 tim_type 0x0


Module in module 2 (present = 0)

Module in module 3 (present = 0)

Module in module 4 (present = 0)

logs-4

N9K# show processes log
VDC Process          PID     Normal-exit  Stack  Core   Log-create-time
--- ---------------  ------  -----------  -----  -----  ---------------
  1 xbar_client      15716             N      N      N Thu May  9 18:01:25 2019
  1 xbar_client      24518             N      N      N Thu May  9 18:03:03 2019
  1 xbar_client      24933             N      N      N Thu May  9 18:04:41 2019
  1 xbar_client      25312             N      N      N Thu May  9 18:06:19 2019

logs-5

N9K# show processes log pid 25312
Service: xbar_client
Description: Xbar Client
Executable: /lc/isan/bin/xbar_client

Started at Thu May  9 18:04:41 2019 (663071 us)
Stopped at Thu May  9 18:06:19 2019 (474464 us)
Uptime: 1 minutes 38 seconds

Start type: SRV_OPTION_RESTART_STATEFUL (24)
Death reason: SYSMGR_DEATH_REASON_FAILURE_HEARTBEAT (9)
Last heartbeat 95.72 secs ago
System image name:
System image version: 7.0(3)I4(7)

PID: 25375
Exit code: signal 6 (no core)

Threads: 25312

CWD: /var/sysmgr/work

RLIMIT_AS:      4294967295

Virtual Memory:

    CODE      100CB000 - 101B3CA8
    DATA      101B4000 - 101B57C8
    BRK       115DD000 - 11771000
    STACK     FF9C7AC0
    TOTAL     618916 KB

Memory Map: 100CB000 xbar_clien 101B4000 xbar_clien D30DE000 mts E30DE000 libmtsdlutils.s E30DF000 libmtsdlutils.s E30E5000 libstathash.s E30E7000 libstathash.s E30E8000 libqosmgr.s
 E30EE000 libqosmgr.s E30EF000 libfm.s E30F7000 libfm.s E30FA000 liburiparse.s E30FD000 liburiparse.s E317A000 libvdc_capability.s E317D000 libvdc_capability.s E317E000 libvdc_mgr_c
mn.s E3181000 libvdc_mgr_cmn.s E3183000 libz.so.1.2. E3198000 libz.so.1.2. E3199000 libifmgr.s E31FF000 libifmgr.s E3248000 libuspace_utils.s E324A000 libuspace_utils.s E324B000 lib
pcm_sdb.s E3272000 libpcm_sdb.s E3278000 libltlmap.s E3284000 libltlmap.s E3287000 libsdwraphist.s E3293000 libsdwraphist.s E3295000 libcmd.s E32BE000 libcmd.s E32C1000 libdleft.s E
32C5000 libdleft.s E32C7000 liburi_map.s E32C9000 liburi_map.s E32CA000 libvsh.s E32E8000 libvsh.s E32EA000 libipfibutils.s E32EE000 libipfibutils.s EAAB5000 libbios.s EAADC000 libb
ios.s EAADE000 libavl.s EAAE1000 libavl.s EAAE2000 libutils_cli_callback.s EAAE3000 libutils_cli_callback.s EAAE4000 libexec.s EAAE7000 libexec.s EAAE8000 libvdb.s EAAEA000 libvdb.s
 EAAF4000 libdll_obj.s EAAF7000 libdll_obj.s EAAF8000 libsysstr.s EAAFA000 libsysstr.s EAAFB000 libsysmgrcmn.s EAB07000 libsysmgrcmn.s EAB08000 libif_index.s EAB47000 libif_index.s
EAB9D000 librt-2.15.s EABA4000 librt-2.15.s EABA5000 librt-2.15.s EABA6000 libbmp.s EAD81000 libbmp.s EAD84000 libeventseq.s EAD98000 libeventseq.s EAD9E000 libsystem_vdc.s EAD9F000
--More--

logs-6

N9K# show system internal xbar event-history lock

1) Event:E_FU_UNLOCK, length:32, at 444695 usecs after Thu May  9 18:09:09 2019
    Status: 0x0
    Gwrap: 0x108cd04c  Cat: 0x0
      Opc:MTS_OPC_LC_INSERTED(1081)
      Msg id: 0X00006B8E
    Lock type: 1
    RID Size: 4
        Val :  0x 2000000


2) Event:E_FU_LOCK, length:32, at 444635 usecs after Thu May  9 18:09:09 2019
    Status: 0x0
    Gwrap: 0x108cd04c  Cat: 0x0
      Opc:MTS_OPC_LC_INSERTED(1081)
      Msg id: 0X00006B8E
    Lock type: 1
    RID Size: 4
        Val :  0x 2000000


3) Event:E_FU_UNLOCK, length:32, at 139624 usecs after Thu May  9 18:09:06 2019
    Status: 0x0
    Gwrap: 0x108cd04c  Cat: 0x0
      Opc:MTS_OPC_LC_INSERTED(1081)
      Msg id: 0X00004EB4
    Lock type: 1
    RID Size: 4
        Val :  0x 2000000


4) Event:E_FU_LOCK, length:32, at 136670 usecs after Thu May  9 18:09:06 2019
    Status: 0x0
    Gwrap: 0x108cd04c  Cat: 0x0
      Opc:MTS_OPC_LC_INSERTED(1081)
      Msg id: 0X00004EB4
    Lock type: 1
    RID Size: 4
        Val :  0x 2000000
  • 2
    There are so many bugs on the Nexus 9k series. I know for a fact, that the first thing Cisco TAC will ask you for is to upgrade to the latest safe harbor 7.0(3)I7(6). We've just had problems ourselves (not same error) and there were no registered bugs or explanation from Cisco. They just told us to upgrade and we haven't had any problems since. – user56700 May 10 at 6:13
  • 1
    Good know! but to get that new version i need to get support :( – Satish May 10 at 11:48
  • 3
    Contact TAC. They may give you a code to download a "fixed" version. (esp. if you hint at removing all Cisco crap because of it.) – Ricky Beam May 10 at 13:45
  • @RickyBeam Thanks for advice, But i doubt they will entertain me without Serial Number or order number :( I am talking to management to get support on it.. – Satish May 10 at 17:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.