How to handle performance degradation deep in your provider's network?

Question

What are some possible ways to detect packet loss deep in a provider's network that is several hops away? With multiple providers peered over BGP on our Internet edge routers, I need to be able to automatically detect packet loss (primarily) and latency (secondarily), and do an interface track or something similar and shut them down so all traffic uses our other providers.

I've seen two problems with using IP SLAs. First, what needs to be measured are at least several hops away (way past their BGP peer), so monitoring anything deep in the provider's network is not a static proposition such as monitoring our links with them (which have been stable); if that provider's links are shutdown, SLAs would still have reachability over another provider's path. Second, doing an ICMP-type monitor doesn't detect the level of packet loss that is usually seen with much larger packets and latency doesn't seem to change significantly.

Is Performance Routing (PfR) the best option here and influencing BGP's localpref? Seems that the Master Controller is a SPoF (Single Point of Failure), so if PfR is the way to go, how can the border routers not be dependent on a single Master Controller? What are two or three other viable options?

The majority and most critical of our traffic is from our outbound HTTP responses.

mellowd · Answer 1 · 2013-06-09T09:59:55.197

7

If using Cisco at the edge, then PfR would indeed be the best option here for the reasons you specified. You can set up master controller redundancy and Cisco shows how on this link: http://www.cisco.com/en/US/docs/solutions/Enterprise/WAN_and_MAN/Transport_diversity/PfR_Master_Controller_Redundancy.html

edited Jun 09 '13 at 09:59

answered Jun 09 '13 at 09:52

mellowd

3,824
19
24

Just read that link: Can you not have a standby controller at another location or is the only topology one with HSRP between controllers? And this seems contrary to what PfR is trying to accomplish if your path from border to controller is degraded: "In the event the PfR border router loses contact with the master controller, the border router stops managing prefixes or applications. In other words, the fail-safe mode of PfR is to remove any routes injected into the IP routing table or BGP table and to stop policy routing of applications if the master controller is unavailable. " – generalnetworkerror Jun 09 '13 at 10:01
Looks like HSRP is the only way. It's a bit silly in my books as Cisco should just be able to engineer PfR to support two master controllers in two separate subnets. You could get a VLL/VPLS to another location and run HSRP over that. Not ideal, but it works. Any 'stretched L2' would work here. Again not ideal, but it would work until Cisco got their act together and allowed two separate controllers. – mellowd Jun 09 '13 at 11:06

score 6 · Accepted Answer · answered Jun 09 '13 at 10:25

6

PfR is indeed one option.

Option which I don't personally have experience in, but I know people using them are BGP optimizers, which are vendor independent as they just look at BGP, measure network, and inject routes to change routing.

Couple options

answered Jun 09 '13 at 10:25

ytti

9,776
42
53

Thanks for the InterNAP FCP option and reminder. We do have INAP peered on these routers, so already have a relationship with them. – generalnetworkerror Jun 10 '13 at 06:21

How to handle performance degradation deep in your provider's network?

2 Answers2