3

We get two 100G links, one from each provider and our aggregate traffic is sub 100G. The rationale is one provider can fail and the other will pick up the slack with zero degradation. Our traffic pattern is fairly symmetrical (i.e. 50G up will be 50G down)

I am using the JunOS platform and wanted to get the communities thought on load balancing strategies. Currently we just take full table from the ISPs and advertise our prefixes with no additional configuration. I have two problems

Ingress

I would suspect that since we are not prepending ingress traffic should be mostly symmetrical. These are both T1 providers and almost anything that "matters" is 1 AS hop away. You would think bandwidth would balance out but it's pretty lopsided in most cases. Why is this the case?

Egress

I look in the routing table (i.e. show route 8.8.8.8, show route 1.1.1.1) and notice all tee destinations are mainly one AS hop away but traffic still seems pretty lopsided.

inet.0: 823020 destinations, 1641766 routes (823020 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

1.1.1.0/24         *[BGP/170] 04:36:45, MED 0, localpref 100, from 4.53.154.145
                      AS path: 3356 13335 I, validation-state: unverified
                    >  to ISPA via et-0/0/1.0
                       to ISPB via et-0/0/3.0
                    [BGP/170] 6d 06:39:30, localpref 100
                      AS path: 1299 13335 I, validation-state: unverified
                    >  to ISPA via et-0/0/1.0

I could probably balance egress by turning on

bgp {
    group isp {
    /* peering info */
    multipath {
        multiple-as
    }
}

policy-options {
    policy-statement LB {
            then {
                load-balance destination-ip-only;
            }
    }
}
routing-options {
    forwarding-table {
        export LB;
    }
}

I am worried this will make troubleshooting "slowness" difficult as things are harder to chase down and really won't be symmetric. Is this a bad thing or is this common place?

Issues and Solutions

The mismatching of ingress\egress traffic is causing me to go into burst on two ISPs vs 1 (ISP A may have high ingress but low egress and ISP B may have high egress but low ingress). What is the communities strategy to solve the above problems? My suggestions

  • Do the suggestion above and enable load sharing between the ISPs
  • Do an active\passive setup where I take ISP B and prepend\depreference it. My fear is this causes unneeded latency as I'm artificially favoring ISP A. I am getting full table. These are two good tier 1 ISPs though and for most things latency is fine.
  • Disaggregate my blocks and somehow advertise half with ISP A preferred and half with ISP preferred. I'd somehow have to local preference the respective groups on a single routing table.

Also why is ISP A showed twice in my 'show route' above

5

Trying to get true load balancing with different providers is a Sisyphean task. You can spend the rest of your career working on it, and never get it right.

These are both T1 providers and almost anything that "matters" is 1 AS hop away. You would think bandwidth would balance out but it's pretty lopsided in most cases. Why is this the case?

BGP always picks one route as the "best." The process is designed to pick one path, even if everything seems the same. Here's Juniper's route selection process:

When everything is well matched (same path length, etc), the selection criteria may come down to something as simple (and arbitrary) as the BGP router id of your peers. There's not a lot you can do about that.

I am worried this will make troubleshooting "slowness" difficult as things are harder to chase down and really won't be symmetric. Is this a bad thing or is this common place?

It will certainly make troubleshooting more difficult, since it will be hard to determine which path a particular flow was used.

While a truly balanced load seems like a great ideal, the effort and configuration required to make it happen probably isn't worth it, especially since you have plenty of bandwidth. I'm sure you have more important things to work on :)

8
  • When you get into these BWs costs come into play. Let's say both ISPs have a 30G commit ISPA is 20\50 in traffic and ISP B is 50/20 in traffic. You now have both ISPs using 50G of traffic. By doing a 'prefer A' failover 'B' you can avoid bursting in B and upping commit in A. Is the prefer A failover B strategy common? or will this needlessly add latency since stuff that should have taken B takes A. ISPs use higher of the two numbrs so having unbalanced up\down traffic artifically makes both ISPs higher Feb 25 at 14:48
  • You can try playing around with local preference/path length on individual prefixes. Be prepared to spend a lot of time on this.
    – Ron Trunk
    Feb 25 at 14:52
  • For my current customer, we use "prefer A, failover B" If, as you say, everything is one AS away, latency probably doesn't matter much, although I know nothing about your business requirements.
    – Ron Trunk
    Feb 25 at 14:55
  • 1
    Also, since these are T1 providers, one hop away can be halfway around the world. there's no way to know without doing a traceroute.
    – Ron Trunk
    Feb 25 at 14:58
  • 1
    For split advertisements, see my comment above.
    – Ron Trunk
    Feb 25 at 14:59
1

I've been doing this since the 90s. The best tips I can give are:

  • use as-path-groups to reduce the local-preference of paths you want to avoid.
  • use caution when increasing local-preference of routes; for example, don't pref-up 3356 174 .* because then you'll choose paths like 3356 174 65000 65000 65000 65000 where network 65000 has tried to reduce their ingress via that path
  • keep a record of your changes; junos makes this trivial

I like to have several as-path-groups to organize my BGP TE, for example:

  • as3356-prefup2 paths I definitely want going to as3356
  • as3356-prefup4 in case need to temporarily override another prefup2
  • as3356-prefdn2 paths to avoid for TE reasons
  • as3356-prefdn4 paths to avoid for bad performance reasons / complaints
  • as3356-prefdn8 temporary overrides
4
  • That only helps the egress piece, not ingress. I still have unbalanced ingress traffic to the point of wrecking my commits (say 50\50 split on egress but 80\20 split on ingress... One ISP is going to burst). Can you post a JUNOS example of what you're doing so I can follow. Feb 25 at 17:26
  • I'll try to edit later and expand my answer with some examples. Indeed, the ingress side is very situational -- your transit providers may support BGP community features to help you with the ingress balance you're looking for. However, to be honest, I think ingress balancing is usually a mistake that decreases network performance while increasing complexity; so I didn't address it in my answer. Feb 25 at 20:09
  • Thanks. Most providers are heavy egress. We are both. Is it possible to do the "prefer A, fail B" for half the prefixes both ingress and egress and "prefer B, fail A" for the other half? I can see how to do it on ingress. Two BGP groups with different prepends. On egress I don't know how I can local pref it based on source IP. Ideally I'd like a symmetric route for tshoot purposes. Being able to split gives me control (customers who complain about ISP A can get shuffled to a prefix that prefers B in and out) Feb 25 at 21:56
  • For egress, if you define half the prefixes as 0/1 and 128/1 (or similar) yes, but you don't need to use such crude methods for egress. You may need to iterate on your as-path-groups, though. For ingress you can do as you describe if your transit providers have the needed BGP community features. What you're looking for is the ability to mark routes as backup routes via announced community. Feb 26 at 13:47
0

Another option is to stop receiving full tables and only get a default route. Then you can use BGP multipath and sort of balance your outbound traffic. Since you can't control the Internet, it's hard to do the same for inbound.

6
  • I can do this anyway w\ full table with what I put above (equal routes will LB) :load-balance destination-ip-only:". The thing is I'm thrashing between ISPs and when things are "slow" it's hard to tshoot. Feb 25 at 17:23
  • Yes, that's the drawback. You have to decide if it's worth it.
    – Ron Trunk
    Feb 25 at 17:52
  • 1
    In your experience is the most "common" approach to do the load balancing and deal with asymmetric traffic or the "prefer A, fail to B" Feb 25 at 19:29
  • I can really only speak about the customers I have worked with. None of them attempt to load balance. But they are all enterprise/government entities, not content providers.
    – Ron Trunk
    Feb 25 at 19:33
  • Is it possible to do the "prefer A, fail B" for half the prefixes both ingress and egress and "prefer B, fail A" for the other half? I can see how to do it on ingress. Two BGP groups with different prepends. On egress I don't know how I can local pref it based on source IP. Ideally I'd like a symmetric route for tshoot purposes. Being able to split gives me control (customers who complain about ISP A can get shuffled to a prefix that prefers B in and out) Feb 25 at 21:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.