Anycast Measurements Used to Highlight Routing Instabilities

Peter Boothe, Randy Bush

This research owes a debt to the NSF via award ANI-0221435, IIJ, University of Oregon, Lucy Lynch, Joel Jaeggli, Mark Kosters, James Hiebert, and many dozens of network ops around the world.

   http://soy.dyndns.org/~peter/projects/research/anycast/nanog

Why do This?

• Verisign presentation Life and Times of J-Root
  • non-trivial routing jitter (Slides 27 through 29)
  • "DO NOT RUN Anycast with Stateful Transport"
• But for almost a decade, there have been reports of successful delivery of stateful services over anycast
• What's going on?

First Things First

• Contacted the folks at Verisign
  • Found out which routes were flapping
• Watched the relevant routes flap using BGPlay
• Designed an experiment to check how common this behavior is

The Experiment

• Sent out emails to *NOG, and got a multitude of hosts around the world to run our collection script
  • Sent a UDP query every 2 seconds to each anycasted root
  • Sent a TCP query every 20 seconds to each anycasted root
  • Results were emailed to a central server
• Query looked like:

  dig @X.root-servers.net. \
hostname.bind chaos txt

Sample Script Output

192.87.109.158
Mon Dec 6 23:00:20 UTC 2004 c UDP jfk1b.c.root-servers.org
Mon Dec 6 23:00:20 UTC 2004 f UDP sfo2b.f.root-servers.org
Mon Dec 6 23:00:20 UTC 2004 i UDP s1.chi
Mon Dec 6 23:00:21 UTC 2004 j UDP jns1-igtld.j.root-servers.net
Mon Dec 6 23:00:21 UTC 2004 k UDP k1.ams-ix
Mon Dec 6 23:00:21 UTC 2004 m UDP M-d1
Mon Dec 6 23:00:23 UTC 2004 c UDP jfk1b.c.root-servers.org
Mon Dec 6 23:00:23 UTC 2004 f UDP sfo2b.f.root-servers.org
Mon Dec 6 23:00:23 UTC 2004 i UDP s1.chi
Mon Dec 6 23:00:23 UTC 2004 j UDP jns2-igtld.j.root-servers.net
Mon Dec 6 23:00:23 UTC 2004 k UDP k1.ams-ix
Mon Dec 6 23:00:23 UTC 2004 m UDP M-d1
Mon Dec 6 23:00:26 UTC 2004 c TCP jfk1b.c.root-servers.org
Mon Dec 6 23:00:26 UTC 2004 f TCP sfo2b.f.root-servers.org
Mon Dec 6 23:00:26 UTC 2004 i TCP s1.chi
Mon Dec 6 23:00:26 UTC 2004 j TCP jns2-igtld.j.root-servers.net
Mon Dec 6 23:00:26 UTC 2004 k TCP k1.ams-ix
Mon Dec 6 23:00:26 UTC 2004 m TCP M-d1
...

Disclaimer

• This is about routing
• This is not about root server performance
• We regexp'd each hostname into a cluster identifier - we only want inter-cluster flapping, not load balancing
• The effects you are about to see are probably caused by:
  • Inter-ISP eBGP
  • Intra-ISP iBGP
  • Intra-ISP IGP (OSPF or IS-IS)

Results (for posterity)

Breaking it Down

• Wide distribution
• Planetlab is more stable
• Internet has a larger variance

Some Sample AS Graphs

Some things to note about the upcoming slides:

• Sometimes the changes all track together (we suspect iBGP or IGP churn)*
• Sometimes they don't (perhaps indicating eBGP churn?)*
• Sometimes they do a little of both

* Rampant supposition and the subject of ongoing investigation

An Example

The Good

The Bad

The Ugly

It Keeps Going...

Now Let's Talk About Failures

• TCP failed more than UDP did (as a percentage)
• We have very little idea what is causing these failures
• Sometimes the failures are Poisson - indicating that the events causing the failures are independent
• Are switches and failures correlated? (Do AS's with a greater amount of switching see more no replies?)

UDP vs. TCP

Failure rates often very distinct

UDP many f-root no-replys	TCP many k-root no-replys
Same AS, same times, very different behaviors

Switches and Failures

Conclusions

• Routing weirdness seems to affect Anycast disproportionately
• PlanetLab is not the same as the internet in terms of routing. In particular, it seems more stable in average
• Most ASs are okay, but when things get bad, they can be very bad
• UDP and TCP probes often had very disparate failure rates
• Your AS's data can be seen at http://limestone.uoregon.edu/~peter

Questions?