How to test a web server before pointing the DNS at it

[fanf]

A large amount of my support work is helping people set up web sites. It's time-consuming because we often have to co-ordinate between three or more groups: typically University IT (me and colleagues), the non-technical owner of the web site, and some commercial web consultancy. And there are often problems, so the co-ordination overhead makes them even slower to fix.

When moving an existing web site, I check that the new web server will work before I update the DNS - it's embarrassing if they have an outage because of an easy-to-avoid cockup, and it's good if we can avoid a panic.

I use a little wrapper around curl --resolve for testing. This makes curl ignore the DNS and talk to the web server I tell it to, but it still uses the new host name when sending the Host: header and TLS SNI and doing certificate verification.

You use the script like:

    curlto <target server> [curl options] <url>

e.g.

    curlto ucam-ac-uk.csi.cam.ac.uk -LI http://some.random.name

This needs a bit of scripting because the curl --resolve option is a faff: you need to explicitly map the URL hostname to all the target IP addresses, and you need to repeat the mapping for both http and https.

Here's the script:

    #!/usr/bin/perl

    use warnings;
    use strict;

    use Net::DNS;

    my $dns = new Net::DNS::Resolver;

    sub addrs {
        my $dn = shift;
        my @a;
        for my $t (qw(A AAAA)) {
            my $r = $dns->query($dn, $t) or next;
            push @a, map $_->address, grep { $_->type eq $t } $r->answer;
        }
        die "curlto: could not resolve $dn\n" unless @a;
        return @a;
    }

    unless (@ARGV > 1) {
        die "usage: curlto <target server> [curl options] <url>\n";
    }

    my $url = $ARGV[-1];
    $url =~ m{^(https?://)?([a-z0-9.-]+)}
        or die "curlto: could not parse hostname in '$url'\n";
    my $name = $2;

    my @addr = shift;
    @addr = addrs @addr unless $addr[0] =~ m{^([0-9.]+|[0-9a-f:]+)$};
    for my $addr (@addr) {
        unshift @ARGV, '--resolv', "$name:80:$addr";
        unshift @ARGV, '--resolv', "$name:443:$addr";
    }

    print "curl @ARGV\n";
    exec 'curl', @ARGV;

Comments

[simont]

This is the kind of thing where I wish there was a more general way to set an environment variable that would override name resolution. HOSTALIASES tries, but doesn't really do what you want, in that it doesn't like (a) aliasing a name of your choice to an IP address rather than another name, or (b) aliasing a qualified name rather than a bare one.

If there was a thing like that that would work sensibly, you could do this same trick independently of browser, rather than being restricted to curl. Then you could use it to test a web site that had never been brought up before and was still expected to have bugs in the dark corners of the UI that you can only get to by a contorted path of link-clicking, cookie-accepting and Javascript-driving in a full GUI browser. Being limited to curl essentially means you can only test the migration of an already-working site to a new server on which the most likely failure mode is not that a dark corner goes wrong but that the entire site forgets to serve you even the simplest page.

Testing before DNS changes

[ewen]

Historically I've done this kind of pre-DNS-change testing by editing /etc/hosts (or equivalent on other systems), and ensuring the system uses /etc/hosts for name resolution first (eg, nsswitch configuration). It's a bit fiddly to do, but has the advantage that because it overrides the system wide resolution of the name it works in pretty much any application (some applications need restarting to see the /etc/hosts change; but most will see it pretty much immediately, if they just defer all name lookups to the system resolution libraries).

I've also sometimes done this with iptables OUTPUT table NAT to redirect the old IP on the system I'm testing to, to the new IP, so that even using the DNS pointing at the old IP will test the new IP/server. But to be sure that's working you do need to watch the logs on the new server. It too has the advantage of being system wide, and thus working in all applications (but is somewhat Linux-as-a-client specific -- I've done something similar on OS X before, using pf features, that might work elsewhere, for a different reason, but it was rather more fiddly).

Knowing how to make curl just pretend the name resolved differently sounds handy for doing quick tests of things like "has the virtual domain actually loaded" and "does it point at the right directory". Thanks for the tip.

Ewen

[(Anonymous)]

I've always thought there should be some kind of URL syntax for this to specify the Host: header sepearately from the IP address (and port) used to connect - something along the lines of "http://www.example.org%localhost:6789/index.html".

While it would be useful for testing websites, I've wanted to use it for both port forwarding (e.g. tunnelling to an intranet site), or for when the DNS is either broken or hasn't propagated yet but I know the IP address.

Chris E