Why does this router fail to use multiple, equal-cost routes from OSPF?

Question

A Brocade router running NetIron 5.2, and a Dynamips emulation running IOS 12.4 both fail to dependably use equal cost paths in what seems to be a very simple configuration. I say "dependably" because I did observe very briefly the expected behavior from the Brocade, but it was after a longish period of inactivity and vanished as soon as I perturbed the system.

In case it is relevant, I am setting up an anycast destination for a stateless network service. This will be a site-specific configuration, with little or no network path length diversity, so the purpose of the anycast capability is mainly for fail-over redundancy and partly for load sharing. My question concerns the load sharing function.

My prototype network looks like this.

R1 <--N1--> Rcore <--N2--> R2

R1 and R2 advertise the anycast destination via OSPF. If Rcore has identical cost to on its N1 and N2 interfaces, and the anycast destination has identical cost at each of the other routers, I would expect to find two equal cost paths in the Rcore routing table. I don't, at least not always, and not even predictably. Why not?

Among other, potentially relevant details are the following.

• Rcore is an ABR connecting N1 and N2, which are in separate NSSA areas, to the backbone (not shown).
• Inspection of the Rcore LSDB confirms that the expected LSAs from R1 and R2 are correctly installed, and contain the anycast destination with the correct metric.

I have considered three possible explanations.

1. I don't understand how to configure and operate ECMP on these platforms. This is likely, and I hope to find enlightenment here that I could not find in the vendor docs.
2. I don't understand the interaction among OSPF Areas, ECMP, anycast, etc. I don't think this is very likely, but I am prepared to learn something anyway.
3. These two vendors implementations suffer from the same bug, or mis-feature, or something. I suppose it could happen.

Edit Added example config and output for Dynamips/Dynagen implementation.

Configs

Router R1

!
version 12.4
!
hostname R1
!
interface Loopback0
 ip address 1.1.1.1 255.255.255.255
 ip ospf cost 1
!
interface FastEthernet0/0
 ip address 10.1.0.2 255.255.255.252
 ip ospf cost 2
!
router ospf 1
 router-id 10.1.0.2
 log-adjacency-changes
 area 0.0.0.1 nssa no-summary
 passive-interface Loopback0
 network 1.1.1.1 0.0.0.0 area 0.0.0.1
 network 10.1.0.0 0.0.0.3 area 0.0.0.1
!
line console 0
 exec-timeout 0 0
!
end

Router R2

!
version 12.4
!
hostname R2
!
interface Loopback0
 ip address 1.1.1.1 255.255.255.255
 ip ospf cost 1
!
interface FastEthernet0/0
 ip address 10.2.0.2 255.255.255.252
 ip ospf cost 2
!
router ospf 1
 router-id 10.2.0.2
 log-adjacency-changes
 area 0.0.0.2 nssa no-summary
 passive-interface Loopback0
 network 1.1.1.1 0.0.0.0 area 0.0.0.2
 network 10.2.0.0 0.0.0.3 area 0.0.0.2
!
line console 0
 exec-timeout 0 0
!
end

Router Rcore

!
version 12.4
!
hostname Rcore
!
interface Loopback0
 ip address 10.0.0.1 255.255.255.255
 ip ospf cost 4
!
interface FastEthernet0/0
 ip address 10.1.0.1 255.255.255.252
 ip ospf cost 2
!
interface FastEthernet0/1
 ip address 10.2.0.1 255.255.255.252
 ip ospf cost 2
!
router ospf 1
 log-adjacency-changes
 area 0.0.0.1 nssa no-summary
 area 0.0.0.2 nssa no-summary
 passive-interface Loopback0
 network 10.0.0.1 0.0.0.0 area 0.0.0.0
 network 10.1.0.0 0.0.0.3 area 0.0.0.1
 network 10.2.0.0 0.0.0.3 area 0.0.0.2
!
line console 0
 exec-timeout 0 0
!
end

Output

Router Rcore

Router LSA from R1

Rcore#sh ip ospf database router 10.1.0.2

            OSPF Router with ID (10.0.0.1) (Process ID 1)

        Router Link States (Area 0.0.0.1)

  LS age: 1618
  Options: (No TOS-capability, DC)
  LS Type: Router Links
  Link State ID: 10.1.0.2
  Advertising Router: 10.1.0.2
  LS Seq Number: 80000002
  Checksum: 0x726F
  Length: 48
  Number of Links: 2

    Link connected to: a Stub Network
     (Link ID) Network/subnet number: 1.1.1.1
     (Link Data) Network Mask: 255.255.255.255
      Number of TOS metrics: 0
       TOS 0 Metrics: 1

    Link connected to: a Transit Network
     (Link ID) Designated Router address: 10.1.0.2
     (Link Data) Router Interface address: 10.1.0.2
      Number of TOS metrics: 0
       TOS 0 Metrics: 2

Router LSA from R2

Rcore#sh ip ospf database router 10.2.0.2

            OSPF Router with ID (10.0.0.1) (Process ID 1)

        Router Link States (Area 0.0.0.2)

  LS age: 1766
  Options: (No TOS-capability, DC)
  LS Type: Router Links
  Link State ID: 10.2.0.2
  Advertising Router: 10.2.0.2
  LS Seq Number: 80000002
  Checksum: 0x8A53
  Length: 48
  Number of Links: 2

    Link connected to: a Stub Network
     (Link ID) Network/subnet number: 1.1.1.1
     (Link Data) Network Mask: 255.255.255.255
      Number of TOS metrics: 0
       TOS 0 Metrics: 1

    Link connected to: a Transit Network
     (Link ID) Designated Router address: 10.2.0.2
     (Link Data) Router Interface address: 10.2.0.2
      Number of TOS metrics: 0
       TOS 0 Metrics: 2

Routing table excerpt

Rcore#sh ip route 1.1.1.1
Routing entry for 1.1.1.1/32
  Known via "ospf 1", distance 110, metric 3, type intra area
  Last update from 10.1.0.2 on FastEthernet0/0, 00:30:54 ago
  Routing Descriptor Blocks:
  * 10.1.0.2, from 10.1.0.2, 00:30:54 ago, via FastEthernet0/0
      Route metric is 3, traffic share count is 1

Answer 1

My first thought would be that OSPF is thinking it's the same LSA instead of simply the same prefix. I don't know much about brocade, but with the LSA outputs from the Cisco box I could probably tell you more.

In general, for Cisco, if the routes are equal in the protocol they will both be presented to the routing table. The routing table, will, by default, install both paths and do ECMP. If the metrics or protocols differ, they will not ECMP. This would also apply to different LSAs for the same prefix.

Answer 2

I am considering whether ECMP makes sense for anycast. Routing should direct anycast to the logically-nearest destination in which one host would be favored over the other. Even in the case where the anycast hosts are equidistant, I would still expect this to remain true. As for ECMP load-sharing, I would think per-destination is impossible with a /32. Perhaps there are load-sharing algorithms that consider the source. Per-packet seems risky, even for DNS. I believe most out-of-the-box anycast solutions run ospf on the host.

Took a bit of searching, but the following excerpt from rfc2178 confirms the cause.

16.8. Equal-cost multipath

The OSPF protocol maintains multiple equal-cost routes to all destinations. This can be seen in the steps used above to calculate the routing table, and in the definition of the routing table structure.

Each one of the multiple routes will be of the same type (intra-area, inter-area, type 1 external or type 2 external), cost, and will have the same associated area. However, each route specifies a separate next hop and Advertising router.

Answer 3

I can't reproduce this in my lab, using 12.4(24). Same topology, RCore area 0 loopback, R1 with lo0 and interface in area 1, R2 with lo0 and interface in area 2 (both NSSA), redistributed route gets picked up for ECMP on Rcore:

Routing entry for 10.20.0.0/16
  Known via "ospf 1", distance 110, metric 20, type NSSA extern 2, forward metric 6477
  Last update from 2.2.2.6 on Serial1/1, 00:02:23 ago
  Routing Descriptor Blocks:
    2.2.2.6, from 2.2.2.21, 00:02:23 ago, via Serial1/1
      Route metric is 20, traffic share count is 1
  * 1.1.2.1, from 1.1.1.1, 00:06:31 ago, via Serial1/0
      Route metric is 20, traffic share count is 1

Possibilities I can think of: Depending on how the route is making it into OSPF, and what OSPF route-type is used, cost to ASBR router may/may not be factored into the route selection process (for ECMP to work, both route cost, and cost to ASBR have to match). Seeing the metric on the actual LSA for both may help.

I was able to reproduce this on IOS by mimic'ing your configuration -- placing the anycast on an interface, and placing the interface into an area. If you do need ECMP, consider redistributing the route into OSPF, as that does appear to work.

Answer 4

Depending on what Brocade hardware you're using (MLX? CER? XMR?), you might want to update the software. There were several bugs fixed through the 5.3 (and 5.4 depending on your hardware) software trees who's symptoms were related to OSPF route costs not being updated properly. I don't know all the criteria for seeing the bugs, but a couple of them were: If the cost was changed on a 1gig port or a loopback interface on a router running OSPF, the change was not processed and the cost was not updated.

I realize you're seeing this on two manufacturers' platforms, so this is more likely to be a configuration issue. I also realize that the old tech-support-punt "update your software" can often be worth nothing more than an eye roll. That said, it might be something to keep in mind.