0

I have an MX960 box that faced an issue as below. Mater RE (RE1) can't show any command invoke-on backup RE (RE0), for example: show version invoke-on all-routing-engines.

{master}

nms@MX960> request routing-engine login other-routing-engine   

connect to address 128.0.0.4: Operation timed out

Trying 10.0.0.4...

re0: Operation timed out

connect to address 128.0.0.4: Operation timed out

Trying 10.0.0.4...

re0: Operation timed out

 

####show database-replication summary

{master}

nms@MX960> show database-replication summary

 

General:

    Graceful Restart           Enabled

    Mastership                 Master

    **Connection                 Down**

    Database                   Available

    Message Queue              Not Ready

 

####show task replication

{master}

nms@MX960> show task replication

        Stateful Replication: Enabled

        RE mode: Master

 

    Protocol                Synchronization Status

    OSPF                    NotStarted           

    BGP                     NotStarted           

    MPLS                    NotStarted           

    RSVP                    NotStarted           

    LDP                     NotStarted  

I checked logs on backup RE (RE0) and got that interface ESW port 13 (interface connects SCB0 and RE0) flapped several times. But then, it is up again, but from RE1, I still couldn't show any command invoke-on backup RE (RE0).

enter image description here

KB33394 https://supportportal.juniper.net/s/article/MX-Internal-interfaces-em0-and-em1-on-MX240-MX480-MX960-explained?language=en_US

From master RE, show chassis routing-engine, I see all REs (RE0 is backup an RE1 is master). But from backup RE, show chassis routing-engine, state of master RE is Present.

**From RE0 (backup RE)**
viettel@KHA0011PRV01> show chassis routing-engine    
Routing Engine status:
  Slot 0:
    Current state                  Backup
    Election priority              Master (default)
    Temperature                 31 degrees C / 87 degrees F
    CPU temperature             30 degrees C / 86 degrees F
    DRAM                      16329 MB (16384 MB installed)
    Memory utilization           9 percent
    5 sec CPU utilization:
      User                       0 percent
      Background                 0 percent
      Kernel                     0 percent
      Interrupt                  0 percent
      Idle                     100 percent
    1 min CPU utilization:
      User                       0 percent
      Background                 0 percent
      Kernel                     0 percent
      Interrupt                  0 percent
      Idle                     100 percent
    5 min CPU utilization:
      User                       0 percent
      Background                 0 percent
      Kernel                     0 percent
      Interrupt                  0 percent
      Idle                     100 percent
    15 min CPU utilization:
      User                       0 percent
      Background                 0 percent
      Kernel                     0 percent
      Interrupt                  0 percent
      Idle                     100 percent
    Model                          RE-S-1800x4
    Serial ID                      9009219924
    Start time                     2022-08-18 11:39:45 ICT
    Uptime                         98 days, 22 hours, 4 minutes, 22 seconds
    Last reboot reason             Router rebooted after a normal shutdown.
    Load averages:                 1 minute   5 minute  15 minute
                                       0.02       0.11       0.14
Routing Engine status:
  Slot 1:
    Current state                  Present

**From RE1 (master RE)**
{master}
nms@KHA0011PRV01> show chassis routing-engine 
Nov 25 09:43:52
Routing Engine status:
  Slot 0:
    Current state                  Backup
    Election priority              Master (default)
    Temperature                 31 degrees C / 87 degrees F
    CPU temperature             30 degrees C / 86 degrees F
    DRAM                      16329 MB (16384 MB installed)
    Memory utilization           9 percent
    5 sec CPU utilization:
      User                       0 percent
      Background                 0 percent
      Kernel                     0 percent
      Interrupt                  0 percent
      Idle                      99 percent
    Model                          RE-S-1800x4
    Serial ID                      9009219924
    Start time                     2022-08-18 11:39:45 ICT
    Uptime                         98 days, 22 hours, 3 minutes, 17 seconds
    Last reboot reason             Router rebooted after a normal shutdown.
    Load averages:                 1 minute   5 minute  15 minute
                                       0.11       0.13       0.15
Routing Engine status:
  Slot 1:
    Current state                  Master
    Election priority              Backup (default)
    Temperature                 32 degrees C / 89 degrees F
    CPU temperature             30 degrees C / 86 degrees F
    DRAM                      16329 MB (16384 MB installed)
    Memory utilization           9 percent
    5 sec CPU utilization:
      User                       0 percent
      Background                 0 percent
      Kernel                     2 percent
      Interrupt                  0 percent
      Idle                      98 percent
    1 min CPU utilization:
      User                       0 percent
      Background                 0 percent
      Kernel                     2 percent
      Interrupt                  0 percent
      Idle                      98 percent
    5 min CPU utilization:
      User                       0 percent
      Background                 0 percent
      Kernel                     2 percent
      Interrupt                  0 percent
      Idle                      98 percent
    15 min CPU utilization:
      User                       0 percent
      Background                 0 percent
      Kernel                     2 percent
      Interrupt                  0 percent
      Idle                      98 percent
    Model                          RE-S-1800x4
    Serial ID                      9009219890
    Start time                     2022-02-10 00:50:35 ICT
    Uptime                         288 days, 8 hours, 53 minutes, 13 seconds
    Last reboot reason             0x1:power cycle/failure
    Load averages:                 1 minute   5 minute  15 minute
                                       0.14       0.23       0.23

After I rebooted RE0, everything OK. Does anyone face the same issue? Pls, give me some points of your view. Thanks.

2
  • I have a slight inkling that the port flapping somehow locked up or crashed the service connection. Have you checked the logs? Do you have any idea what may have caused the flapping?
    – Zac67
    Commented Nov 25, 2022 at 9:29
  • Did any answer help you? If so, you should accept the answer so that the question does not keep popping up forever, looking for an answer. Alternatively, you could post and accept your own answer.
    – Ron Maupin
    Commented Dec 15, 2022 at 22:28

2 Answers 2

0

I checked log chassisd of RE0 (backup RE), no special event happened before port 13 flapped. port 13 flap many times. But in the below time, I observed strange logs, I pushed continuously.

Nov 20 12:33:55  ch_gencfg_chassis_startup_time_handler: master_re: false, GENCFG_CHASSIS_STARTUP_TIME, minor_type: 8
Nov 20 12:33:55  ch_gencfg_chassis_startup_time_blob_get: chassis startup time from kernel is 1644429032.323756
Nov 20 12:33:55  ch_gencfg_chassis_startup_time_handler: wrote hw.chassis.startup_time as 1644429032.323756
Nov 20 13:33:53  ch_gencfg_chassis_startup_time_handler: master_re: false, GENCFG_CHASSIS_STARTUP_TIME, minor_type: 8
Nov 20 13:33:53  ch_gencfg_chassis_startup_time_blob_get: chassis startup time from kernel is 1644429032.323753
Nov 20 13:33:53  ch_gencfg_chassis_startup_time_handler: wrote hw.chassis.startup_time as 1644429032.323753
Nov 20 13:48:25  send: red alarm set, device Routing Engine 0, reason Host 0 em0 : Ethernet Link to PFEs Down
Nov 20 13:48:30  acb_update_local_esw_status: ESW port 13 (connected to RE-GigE) link down
Nov 20 13:48:30  send: red alarm clear, device Routing Engine 0, reason Host 0 em0 : Ethernet Link to PFEs Down
Nov 20 13:48:33  fru_is_present: out of range slot 1 for 
Nov 20 13:48:43  acb_update_local_esw_status: ESW port 13 (connected to RE-GigE) link up
Nov 20 13:48:46  fru_is_present: out of range slot 1 for 
Nov 20 13:48:59  fru_is_present: out of range slot 1 for 
Nov 20 13:49:13  fru_is_present: out of range slot 1 for 
Nov 20 13:49:26  fru_is_present: out of range slot 1 for 

The log "fru_is_present: out of range slot 1 for" pushed continuously until we rebooted RE0. After RE0 online normally, we only observed logs like below:

Nov 28 12:33:54  ch_gencfg_chassis_startup_time_handler: master_re: false, GENCFG_CHASSIS_STARTUP_TIME, minor_type: 8
Nov 28 12:33:54  ch_gencfg_chassis_startup_time_blob_get: chassis startup time from kernel is 1644429032.323070
Nov 28 12:33:54  ch_gencfg_chassis_startup_time_handler: wrote hw.chassis.startup_time as 1644429032.323070
Nov 28 13:33:53  ch_gencfg_chassis_startup_time_handler: master_re: false, GENCFG_CHASSIS_STARTUP_TIME, minor_type: 8
Nov 28 13:33:53  ch_gencfg_chassis_startup_time_blob_get: chassis startup time from kernel is 1644429032.323067
Nov 28 13:33:53  ch_gencfg_chassis_startup_time_handler: wrote hw.chassis.startup_time as 1644429032.323067
Nov 28 14:33:52  ch_gencfg_chassis_startup_time_handler: master_re: false, GENCFG_CHASSIS_STARTUP_TIME, minor_type: 8
Nov 28 14:33:52  ch_gencfg_chassis_startup_time_blob_get: chassis startup time from kernel is 1644429032.323064
Nov 28 14:33:52  ch_gencfg_chassis_startup_time_handler: wrote hw.chassis.startup_time as 1644429032.323064
0

That looks like some kind of software bug. I would open a case with the support group for your equipment.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.