Multihoming with two 'good' providers - load sharing strategy?

Question

We get two 100G links, one from each provider and our aggregate traffic is sub 100G. The rationale is one provider can fail and the other will pick up the slack with zero degradation. Our traffic pattern is fairly symmetrical (i.e. 50G up will be 50G down)

I am using the JunOS platform and wanted to get the communities thought on load balancing strategies. Currently we just take full table from the ISPs and advertise our prefixes with no additional configuration. I have two problems

Ingress

I would suspect that since we are not prepending ingress traffic should be mostly symmetrical. These are both T1 providers and almost anything that "matters" is 1 AS hop away. You would think bandwidth would balance out but it's pretty lopsided in most cases. Why is this the case?

Egress

I look in the routing table (i.e. show route 8.8.8.8, show route 1.1.1.1) and notice all tee destinations are mainly one AS hop away but traffic still seems pretty lopsided.

inet.0: 823020 destinations, 1641766 routes (823020 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

1.1.1.0/24         *[BGP/170] 04:36:45, MED 0, localpref 100, from 4.53.154.145
                      AS path: 3356 13335 I, validation-state: unverified
                    >  to ISPA via et-0/0/1.0
                       to ISPB via et-0/0/3.0
                    [BGP/170] 6d 06:39:30, localpref 100
                      AS path: 1299 13335 I, validation-state: unverified
                    >  to ISPA via et-0/0/1.0

I could probably balance egress by turning on

bgp {
    group isp {
    /* peering info */
    multipath {
        multiple-as
    }
}

policy-options {
    policy-statement LB {
            then {
                load-balance destination-ip-only;
            }
    }
}
routing-options {
    forwarding-table {
        export LB;
    }
}

I am worried this will make troubleshooting "slowness" difficult as things are harder to chase down and really won't be symmetric. Is this a bad thing or is this common place?

Issues and Solutions

The mismatching of ingress\egress traffic is causing me to go into burst on two ISPs vs 1 (ISP A may have high ingress but low egress and ISP B may have high egress but low ingress). What is the communities strategy to solve the above problems? My suggestions

• Do the suggestion above and enable load sharing between the ISPs
• Do an active\passive setup where I take ISP B and prepend\depreference it. My fear is this causes unneeded latency as I'm artificially favoring ISP A. I am getting full table. These are two good tier 1 ISPs though and for most things latency is fine.
• Disaggregate my blocks and somehow advertise half with ISP A preferred and half with ISP preferred. I'd somehow have to local preference the respective groups on a single routing table.

Also why is ISP A showed twice in my 'show route' above

Answer 1

Trying to get true load balancing with different providers is a Sisyphean task. You can spend the rest of your career working on it, and never get it right.

These are both T1 providers and almost anything that "matters" is 1 AS hop away. You would think bandwidth would balance out but it's pretty lopsided in most cases. Why is this the case?

BGP always picks one route as the "best." The process is designed to pick one path, even if everything seems the same. Here's Juniper's route selection process:

When everything is well matched (same path length, etc), the selection criteria may come down to something as simple (and arbitrary) as the BGP router id of your peers. There's not a lot you can do about that.

I am worried this will make troubleshooting "slowness" difficult as things are harder to chase down and really won't be symmetric. Is this a bad thing or is this common place?

It will certainly make troubleshooting more difficult, since it will be hard to determine which path a particular flow was used.

While a truly balanced load seems like a great ideal, the effort and configuration required to make it happen probably isn't worth it, especially since you have plenty of bandwidth. I'm sure you have more important things to work on :)

Answer 2

I've been doing this since the 90s. The best tips I can give are:

• use as-path-groups to reduce the local-preference of paths you want to avoid.
• use caution when increasing local-preference of routes; for example, don't pref-up 3356 174 .* because then you'll choose paths like 3356 174 65000 65000 65000 65000 where network 65000 has tried to reduce their ingress via that path
• keep a record of your changes; junos makes this trivial

I like to have several as-path-groups to organize my BGP TE, for example:

• as3356-prefup2 paths I definitely want going to as3356
• as3356-prefup4 in case need to temporarily override another prefup2
• as3356-prefdn2 paths to avoid for TE reasons
• as3356-prefdn4 paths to avoid for bad performance reasons / complaints
• as3356-prefdn8 temporary overrides

Answer 3

Another option is to stop receiving full tables and only get a default route. Then you can use BGP multipath and sort of balance your outbound traffic. Since you can't control the Internet, it's hard to do the same for inbound.