For some time, we've managed DNS with a combination of geographic locations, using zones, and A records, that can give a round robin result for an area, to get the closest result for an HTTP request on the CDN. We recently exceeded the 512 byte limit on DNS answers that is imposed by default on Cisco ASA firewalls, with the result being a brief CDN outage. When you consider that DNSSEC has been sending large answers for eight years, and eDNS timeouts should be handled by lookup servers gracefully, with fallback to 512 octet DNS udp size, it's strange that Cisco still ships with this default setting.
So what have we learned? We can't fix bind9 timeout handling at ISPs, nor can we request that Cisco change their ASA stock setting to not switch to tcp for larger DNS responses. Well we could, but Cisco is the Maytag of network appliance companies. Yeah, the bored service counter guy works there. So for now, we better keep our DNS packets not one byte larger than 512 (it was one byte too big).
We think a full anycast DNS setup will solve this once and for all and will keep you posted on progress on this front. We are also excited about a large ZFS storage system we will be bringing online at Pullman, our recent upgrades in the US for the CDN, and major upgrades that are coming in Europe in January for VOD and streaming customers.