2

We have a ME-3600 we're seeing a strange issue with. The device has an SVI which is used within a full-mesh metro ethernet network. The SVI is used for management and is also the peering source for the routing protocols. For no (apparent) reason the SVI is going down, it is only tied to one interface and we are not seeing the physical interface going down. We've tried pinning the SVI to another interface to keep it up, which would keep the SVI up but we still would see all of the routing protocols bounce.

The device:

problemchild uptime is 5 weeks, 2 days, 11 hours, 47 minutes
System returned to ROM by power-on
System restarted at 19:01:24 MST Fri Feb 13 2015
System image file is "flash:/me360x-universalk9-mz.153-3.S1a/me360x-universalk9-mz.153-3.S1a.bin"
Last reload reason: Reload Command

License Level: AdvancedMetroIPAccess
License Type: Permanent
Next reload license Level: AdvancedMetroIPAccess

cisco ME-3600X-24FS-M (PowerPC8572) processor (revision A0) with 983040K/65528K bytes of memory.

Relevant log messages:

Mar 19 20:51:32.090 MDT: %LDP-5-GR: GR session 10.255.10.63:0 (inst 93): completed graceful recovery
Mar 19 20:51:32.102 MDT: %LDP-5-GR: GR session 10.255.10.47:0 (inst 94): completed graceful recovery
Mar 19 20:51:32.102 MDT: %LDP-5-GR: GR session 10.255.10.50:0 (inst 95): completed graceful recovery
Mar 20 03:28:22.614 MDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan105, changed state to down
Mar 20 03:28:22.630 MDT: %OSPF-5-ADJCHG: Process 1, Nbr 10.255.0.17 on Vlan105 from 2WAY to DOWN, Neighbor Down: Interface down or detached
Mar 20 03:28:22.630 MDT: %OSPF-5-ADJCHG: Process 1, Nbr 10.255.0.33 on Vlan105 from FULL to DOWN, Neighbor Down: Interface down or detached
Mar 20 03:28:22.634 MDT: %OSPF-5-ADJCHG: Process 1, Nbr 10.255.0.49 on Vlan105 from FULL to DOWN, Neighbor Down: Interface down or detached
....continued....
Mar 20 03:28:50.091 MDT: %BGP_SESSION-5-ADJCHANGE: neighbor 10.255.0.2 VPNv4 Unicast topology base removed from session  BGP Notification sent
Mar 20 03:28:52.815 MDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan105, changed state to up
Mar 20 03:28:57.611 MDT: %OSPF-5-ADJCHG: Process 1, Nbr 10.255.0.33 on Vlan105 from LOADING to FULL, Loading Done
Mar 20 03:28:57.875 MDT: %OSPF-5-ADJCHG: Process 1, Nbr 10.255.0.49 on Vlan105 from LOADING to FULL, Loading Done
Mar 20 03:29:08.727 MDT: %LDP-5-NBRCHG: LDP Neighbor 10.255.0.33:0 (1) is UP
Mar 20 03:29:08.727 MDT: %LDP-5-NBRCHG: LDP Neighbor 10.255.0.49:0 (2) is UP
Mar 20 03:29:08.727 MDT: %LDP-5-NBRCHG: LDP Neighbor 10.255.0.17:0 (3) is UP

Interface Details:

problemchild#sh vlan id 105

VLAN Name                             Status    Ports
---- -------------------------------- --------- -------------------------------
105  xxxxxxxxx                         active    Gi0/24


problemchild#sh run int gi0/24
interface GigabitEthernet0/24
 description metroE circuit
 switchport trunk allowed vlan 1,105,144,145,170
 switchport mode trunk
 mtu 1522
 bandwidth 40000
 service-policy output xxx-OUT
end

problemchild#sh run int vl105
interface Vlan105
 description EVC mgmt interface
 mtu 1522
 bandwidth 40000
 ip address 10.15.0.22 255.255.254.0
 ip ospf message-digest-key 1 md5 xxxx
 ip ospf priority 0
 ip ospf mtu-ignore
 mpls ip
end

problemchild#sh int gi0/24
GigabitEthernet0/24 is up, line protocol is up (connected)
  Hardware is Gigabit Ethernet, address is 24b6.5732.2098 (bia 24b6.5732.2098)
  Description: metroE circuit
  MTU 1522 bytes, BW 40000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 76/255, rxload 32/255
  Encapsulation ARPA, loopback not set
  Keepalive not set
  Full-duplex, 1000Mb/s, media type is SX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 00:00:00, output hang never
  Last clearing of "show interface" counters 5w2d
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 1785964
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 5020000 bits/sec, 1398 packets/sec
  5 minute output rate 12010000 bits/sec, 2241 packets/sec
     1820442136 packets input, 922204069899 bytes, 0 no buffer
     Received 58765981 broadcasts (58764761 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 58764761 multicast, 0 pause input
     0 input packets with dribble condition detected
     1422024326 packets output, 852355101625 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

The output policy-map:

problemchild#sh policy-map int gi0/24
 GigabitEthernet0/24

  Service-policy output: xxx-OUT

    Class-map: class-default (match-any)
      1402564353 packets, 839023628650 bytes
      5 minute offered rate 3498000 bps, drop rate 0000 bps
      Match: any
  Traffic Shaping
    Average Rate Traffic Shaping
    Shape 40000 (kbps)
      Output Queue:
        Default Queue-limit 49152 bytes
        Tail Packets Drop: 418017
        Tail Bytes Drop: 373661094

      Service-policy : All_Agency_CBWFQ

        Class-map: xxxx_Q1 (match-any)
          0 packets, 0 bytes
          5 minute offered rate 0000 bps, drop rate 0000 bps
          Match:  precedence 5
          Match:  dscp cs5 (40) 41  42  43  44  45  ef (46) 47
          police:
            cir 9950000 bps, bc 310937 bytes
            conform-action transmit
            exceed-action drop
          conform: 0 (packets) 0 (bytes)
          exceed: 0 (packets) 0 (bytes)
          conform: 0 bps, exceed: 0 bps
          Strict Priority
          Queue-limit 32000 bytes
          Queue-limit current-queue-depth 0 bytes
              Output Queue:
                Default Queue-limit 49152 bytes
                Tail Packets Drop: 0
                Tail Bytes Drop: 0

        Class-map: xxxx_Q2 (match-any)
          0 packets, 0 bytes
          5 minute offered rate 0000 bps, drop rate 0000 bps
          Match:  precedence 4
          Match:  precedence 6
          Match:  precedence 7
          Match:  dscp cs4 (32) 33  af41 (34) 35  af42 (36) 37  af43 (38) 39
          Match:  dscp cs6 (48) 49  50  51  52  53  54  55
          Match:  dscp cs7 (56) 57  58  59  60  61  62  63
          Bandwidth 8950 (kbps)
          Queue-limit 32000 bytes
          Exp-weighted-constant 9
          Random_detect prec
          4 6 7         28000  32000  bytes 10


          Queue-limit current-queue-depth 0 bytes
              Output Queue:
                Default Queue-limit 49152 bytes
                Tail Packets Drop: 0
                Tail Bytes Drop: 0

        Class-map: xxxx_Q3 (match-any)
          1022048 packets, 301438609 bytes
          5 minute offered rate 0000 bps, drop rate 0000 bps
          Match:  precedence 2
          Match:  precedence 3
          Match:  dscp cs2 (16) 17  af21 (18) 19  af22 (20) 21  af23 (22) 23
          Match:  dscp cs3 (24) 25  af31 (26) 27  af32 (28) 29  af33 (30) 31
          Bandwidth 8950 (kbps)
          Queue-limit 32000 bytes
          Exp-weighted-constant 9
          Random_detect prec
          2 3   28000  32000  bytes 10

          Queue-limit current-queue-depth 0 bytes
              Output Queue:
                Default Queue-limit 49152 bytes
                Tail Packets Drop: 0
                Tail Bytes Drop: 0

        Class-map: xxxx_Q4 (match-any)
          779962 packets, 76289466 bytes
          5 minute offered rate 0000 bps, drop rate 0000 bps
          Match:  precedence 0
          Match:  precedence 1
          Match:  dscp default (0) 1  2  3  4  5  6  7
          Match:  dscp cs1 (8) 9  af11 (10) 11  af12 (12) 13  af13 (14) 15
          Bandwidth 11950 (kbps)
          Queue-limit 64000 bytes
          Exp-weighted-constant 9
          Random_detect prec
          0 1   58000  64000  bytes 10

          Queue-limit current-queue-depth 0 bytes
              Output Queue:
                Default Queue-limit 49152 bytes
                Tail Packets Drop: 0
                Tail Bytes Drop: 0

        Class-map: class-default (match-any)
          1400762343 packets, 838645900575 bytes
          5 minute offered rate 3496000 bps, drop rate 0000 bps
          Match: any
  • Please enable "logging event link-status" on gi0/24 to rule out an interface going down. By default a physical interface going down is not logged, unless you have it enabled globally. – dk1 Mar 24 '15 at 0:12
  • I don't believe it was set globally, I found it on another interface so I added it to the gi0/24. I would have thought the interface resets would have incremented as well if it was going down? Also why would it log the SVI going down and not the physical link? – Robert Mar 24 '15 at 13:48
  • Just an update after I added the "logging event link-status" to gi0/24: only the SVI is going down, the physical interface is still staying up. – Robert Mar 25 '15 at 18:22
  • 1
    Is it possible that STP is moving gi0/24 into the blocking state? If it goes into the blocking state and since that is the only interface in V105 it would bring down the SVI. – matak Apr 23 '15 at 19:39
  • Can you try moving to using Loopbacks for LDP/iBGP bindings and using the physical interface for OSPF (IGP) connectivity only so you can then seperate the control plane protocols, redistributing the loopback IP into the IGP. Another level of seperation like this may reveal something extra. Also what debugging have you captured? (debug ip ospf, mpls, ldp, bgp etc) and also can you run a SPAN? Having said that about SPANs they don't work very well on ME3600's, I would highly recommend upgrading your IOS to 15.3(3)S3. – jwbensley Apr 24 '15 at 22:40
1

I noticed that you've got exactly 30 seconds from the time the link goes down until it comes back up again. OSPF Hellos are 30 seconds default on a non-broadcast link. What's your hold timer set to?

| improve this answer | |
  • It is not set, but it is a broadcast network type. Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5 – Robert Mar 24 '15 at 16:25
  • Is the other end of that connection set to the same settings for OSPF? It seems like it could be flapping. I only saw the link go down and the time until it came back up, but is it going down at regular intervals as well or is it more sporadic? – Brandon West Mar 24 '15 at 16:53
  • I meant to add that you and I are connected on LinkedIn. :) Small world! – Brandon West Mar 24 '15 at 16:55
  • It's on a multipoint metroE EVC, roughly 30 neighbors. It's a fairly random/sporadic but it is happening almost every 24 hours. It's not flapping (at least the SVI isn't). Just that one occurrence at random times. We've got a TAC case open, but they're scratching their heads as well. – Robert Mar 24 '15 at 16:56
  • 1
    I've had something similar happen (while not on the scale you are using) at a project in California. Ended up being a media converter from Eth to Fiber boogering up random bits in the data that was being sent/received. It would occasionally corrupt MAC entries (and subsequently flood the unknown MACs) and the FCS discard counter would burst up and the link would go down. Once that corrected, it would resume and would never happen in predictable intervals. Maybe it could be something around there to look at. – Brandon West Mar 24 '15 at 17:04
0

You can configure the "no auto-state" command under the vlan interface to disable the up/down feature of SVIs.

interface vlan105
 no auto-state

I hope this solves it. :)

Cheers,

H

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.