How to test a web server before pointing the DNS at it

[fanf]

A large amount of my support work is helping people set up web sites. It's time-consuming because we often have to co-ordinate between three or more groups: typically University IT (me and colleagues), the non-technical owner of the web site, and some commercial web consultancy. And there are often problems, so the co-ordination overhead makes them even slower to fix.

When moving an existing web site, I check that the new web server will work before I update the DNS - it's embarrassing if they have an outage because of an easy-to-avoid cockup, and it's good if we can avoid a panic.

I use a little wrapper around curl --resolve for testing. This makes curl ignore the DNS and talk to the web server I tell it to, but it still uses the new host name when sending the Host: header and TLS SNI and doing certificate verification.

You use the script like:

    curlto <target server> [curl options] <url>

e.g.

    curlto ucam-ac-uk.csi.cam.ac.uk -LI http://some.random.name

This needs a bit of scripting because the curl --resolve option is a faff: you need to explicitly map the URL hostname to all the target IP addresses, and you need to repeat the mapping for both http and https.

Here's the script:

    #!/usr/bin/perl

    use warnings;
    use strict;

    use Net::DNS;

    my $dns = new Net::DNS::Resolver;

    sub addrs {
        my $dn = shift;
        my @a;
        for my $t (qw(A AAAA)) {
            my $r = $dns->query($dn, $t) or next;
            push @a, map $_->address, grep { $_->type eq $t } $r->answer;
        }
        die "curlto: could not resolve $dn\n" unless @a;
        return @a;
    }

    unless (@ARGV > 1) {
        die "usage: curlto <target server> [curl options] <url>\n";
    }

    my $url = $ARGV[-1];
    $url =~ m{^(https?://)?([a-z0-9.-]+)}
        or die "curlto: could not parse hostname in '$url'\n";
    my $name = $2;

    my @addr = shift;
    @addr = addrs @addr unless $addr[0] =~ m{^([0-9.]+|[0-9a-f:]+)$};
    for my $addr (@addr) {
        unshift @ARGV, '--resolv', "$name:80:$addr";
        unshift @ARGV, '--resolv', "$name:443:$addr";
    }

    print "curl @ARGV\n";
    exec 'curl', @ARGV;

Comments

[simont]

This is the kind of thing where I wish there was a more general way to set an environment variable that would override name resolution. HOSTALIASES tries, but doesn't really do what you want, in that it doesn't like (a) aliasing a name of your choice to an IP address rather than another name, or (b) aliasing a qualified name rather than a bare one.

If there was a thing like that that would work sensibly, you could do this same trick independently of browser, rather than being restricted to curl. Then you could use it to test a web site that had never been brought up before and was still expected to have bugs in the dark corners of the UI that you can only get to by a contorted path of link-clicking, cookie-accepting and Javascript-driving in a full GUI browser. Being limited to curl essentially means you can only test the migration of an already-working site to a new server on which the most likely failure mode is not that a dark corner goes wrong but that the entire site forgets to serve you even the simplest page.