Feb. 18th, 2016

fanf: (dotat)

tl;dr - you can use my program adns-masterfile to pre-heat a DNS cache as part of a planned failover process.

adns

One of my favourite DNS tools is GNU adns, an easy-to-use asynchronous stub resolver library.

I started using adns in 1999 when it was still very new. At that time I was working on web servers at Demon Internet. We had a daily log processing job which translated IP addresses in web access logs to host names, and this was getting close to taking more than a day to process a day of logs. I fixed this problem very nicely with adns, and my code is included as an example adns client program.

Usually when I want to do some bulk DNS queries, I end up doing a clone-and-hack of adnslogres. Although adns comes with a very capable general-purpose program called adnshost, it doesn't have any backpressure to control the number of outstanding queries. So if you try to use it for bulk queries, you will overwhelm the server with traffic and most of the queries will time out and fail. So, although my normal approach would be to massage the data with an ad-hoc Perl script so I can pipe it into a general-purpose program, with adns I usually end up writing special-purpose C programs which have their own concurrency control.

cache preheating

In our DNS server setup we use keepalived to do automatic failover of the recursive server IP addresses. I wrote about our keepalived configuration last year, explaining how we can control which servers are live, which are standby, and which are not part of the failover pool.

The basic process for config changes, patches, upgrades, etc., is to take a stanby machine out of the pool, apply the changes, test them, then shuffle the pool so the next machine can be patched. Patching all of the servers requires one blip of a few seconds for each live service address.

However, after moving a service address, the new server has a cold cache, so (from the point of view of the user) a planned failover looks like a sudden drop in DNS server performance.

It would be nice if we could pre-heat a server's cache before making it live, to minimize the impact of planned maintenance.

rndc dumpdb

BIND can be told to dump its cache, which it does in almost-standard textual DNS master file format. This is ideal source data for cache preheating: we could get one of the live servers to dump its cache, then use the dump to make lots of DNS queries against a standby server to warm it up.

The named_dump.db file uses a number of slightly awkward master file features: it omits owner names and other repeated fields when it can, and it uses () to split records across multiple lines. So it needs a bit more intelligence to parse than just extracting a field from a web log line!

lex

I hadn't used lex for years and years before last month. I had almost forgotten how handy it is, although it does have some very weird features. (Its treatment of leading whitespace is almost as special as tab characters in make files.)

But, if you have a problem whose solution involves regular expressions and a C library, it's hard to do better than lex. It makes C feel almost like a scripting language!

(According to the Jargon File, Perl was described as a "Swiss-Army chainsaw" because it is so much more powerful than Lex, which had been described as the Swiss-Army knife of Unix. I can see why!)

adns-masterfile

So last month, I used adns and lex to write a program called adns-masterfile which parses a DNS master file (in BIND named_dump.db flavour) and makes queries for all the records it can. It's a reasonably handy way for pre-heating a cache as part of a planned server swap.

I used it yesterday when I was patching and rebooting everything to deal with the glibc CVE-2015-7547 vulnerability.

performance

Our servers produce a named_dump.db file of about 5.3 million lines (175 MB). From this adns-masterfile makes about 2.6 million queries, which takes about 10 minutes with a concurrency limit of 5000.

Page generated Jun. 29th, 2017 03:57 pm
Powered by Dreamwidth Studios