What are some possible ways to detect packet loss deep in a provider's network that is several hops away? With multiple providers peered over BGP on our Internet edge routers, I need to be able to automatically detect packet loss (primarily) and latency (secondarily), and do an interface track or something similar and shut them down so all traffic uses our other providers.
I've seen two problems with using IP SLAs. First, what needs to be measured are at least several hops away (way past their BGP peer), so monitoring anything deep in the provider's network is not a static proposition such as monitoring our links with them (which have been stable); if that provider's links are shutdown, SLAs would still have reachability over another provider's path. Second, doing an ICMP-type monitor doesn't detect the level of packet loss that is usually seen with much larger packets and latency doesn't seem to change significantly.
Is Performance Routing (PfR) the best option here and influencing BGP's localpref? Seems that the Master Controller is a SPoF (Single Point of Failure), so if PfR is the way to go, how can the border routers not be dependent on a single Master Controller? What are two or three other viable options?
The majority and most critical of our traffic is from our outbound HTTP responses.