Amsterdam day 1

During IETF 101 I wrote up the day's activity on the way home on the train, which worked really well. My vague plan of doing the same in Amsterdam might be less successful because it's less easy to hammer out notes while walking across the city.

Some semi-chronological notes ... I'm going to use the workshop programme as a starting point, but I'm not going to write particularly about the contents of the talks (see the workshop web site for that) but more about what I found relevant to me and the follow-up hallway discussions.

Morning

The first half of the morning was CENTR members only, so I went to get a few bits from the local supermarket via the "I Amsterdam" sign before heading to the conference hotel.

The second half of the morning was DNS-OARC business. It's a small organization that provides technical infrastructure, so most of the presentation time was about technical matters. The last sessopn before lunch was Matt Pounsett talking about sysadmin matters. He's impressively enthusiastic about cleaning up a mountain of technical debt. We had a chat at the evening social and I tried to convince him that Debian without systemd is much less painful than the Devuan he has been struggling with.

Jerry Lundstrom talked about his software development projects. A couple of these might be more useful to me than I was previously aware:

• drool - DNS replay tool. I have a crappy dnsmirror script which I wrote after my previous test procedure failed to expose CVE-2018-5737 because I wasn't repeating queries.

  drool respdiff sounds like it might be a lot of what I need to automate my testing procedures between deploying to staging and production.

• dnsjit - suite of DNS-related tools in Lua. If this has the right facilities then I might be able to delete the horrid DNS packet parsing code from doh101.

Cloudflare DoH and DoT

Ólafur Guðmundsson said a few noteworthy things:

• Cloudflare's DoT implementation is built-in to Knot Resolver. They are heavy users of my qp trie data structure, and at the evening social Petr Špaček told me they are planning to merge the knot-reolver-specific fixes with my beer festival COW work.

• Their DoH implementation uses Lua scripting in NGINX which sounds eerily familiar :-) (Oversimplifying to the point of being wrong, Cloudflare's web front end is basically OpenResty.)

• He mentioned a problem with hitting quotas on the number of http2 channels opened by Firefox clients, which I need to double check.

• Cloudflare are actively working on DoT for recursive to authoritative servers. Sadly, although the IETF DNS Privacy working group has been discussing this recently, there hasn't been much comment from people who are doing practical work. Cloudflare likes the advantages of holding persistent connections from their big resolvers to popular authoritative servers, which is basically optimizing for a more centralized web. It's the modern version of leased lines between mainframes.

I submitted my DoH / DoT lightning talk (2 slides!) to the program committee since it includes stuff which Ólafur didn't mention that is relevant to other recursive server operators.

ANAME

I merged my revamped draft into Evan Hunt's aname repository and collaboration is happening. I've talked about it with at least one big DNS company who have old proprietary ANAME-like facilities, and they are keen to use standardization as a push towards removing unused features and cleaning up other technical debt. I've put some "mighty weasel words" in the draft, i.e. stealing the "as if" idea from C, with the idea that it gives implementers enough freedom to make meaningful zone file portability possible, provided the zone is only relying on some common subset of features.

Other small notes

Matt Larson (ICANN) - rollover was a "pleasant non-event"

• several ICANN staff are at DNS-OARC, so they travelled to NL before the rollover, and used a conference room at NLnet Labs as an ad-hoc mission control room for the rollover

Matt Weinberg (Verisign) - bad trust anchor telemetry signal investigation - culprit was Monero! I wonder if this was what caused Cambridge's similar weirdness back in June.

Duane Wessels (Verisign) talked about abolishing cross-TLD glue. Gradually the registry data model becomes less insane! [fanf's rant censored]

Jaromír Talíř (CZ.NIC) on TLD algorithm rollovers. Don't fear ECDSA P256! I definitely want to do this for Cambridge's zones.

Bert Hubert (PowerDNS) talked about the DNS Camel and his Hello DNS effort to make the RFCs easier to understand. He's very pleased with the name compression logic he worked out for tdns, a teaching DNS server. I want to have a proper look at it...

Amsterdam day 2

Up at the crack of dawn for the second half of the DNS-OARC workshop. (See the timetable for links to slides etc.) The coffee I bought yesterday morning made a few satisfactory cups to help me get started.

Before leaving the restaurant this evening I mentioned writing my notes to Dave Knight, who said his approach is to incrementally add to an email as the week goes on. I kind of like my daily reviews for remembering interesting side conversations, which are the major advantage of the value of attending these events in person.

DoT / DoH

Sara Dickinson of Sinodun did a really good talk on the consequences of DNS encryption, with a very insightful analysis of the implications for how this might change the architectural relationships between the web and the DNS.

DNS operators should read RFC 8404 on "Effects of Pervasive Encryption on Operators". (I have not read it yet.)

Sara encouraged operators to implement DoT and DoH on their resolvers.

My lightning talk on DoT and DoH at Cambridge was basically a few (very small) numbers to give operators an idea of what they can expect if they actually do this. I'm going to submit the same talk for the RIPE lightning talks session later this week.

I had some good conversations with Baptiste Jonglez (who is doing a PhD at Univ. Grenoble Alpes) and with Sara about DoT performance measurements. At the moment BIND doesn't collect statistics that allow me to know interesting things about DoT usage like DoT query rate and timing of queries within a connection. (The latter is useful for setting connection idle timeouts.) Something to add to the todo list...

CNAME at apex

Ondřej Surý of ISC.org talked about some experiments to find out how much actually breaks in practice if you put a CNAME and other data at a zone apex. Many resolvers break, but surprisingly many resolvers kind of work.

Interestingly, CNAME+DNAME at the same name is pretty close to working. This has been discussed in the past as "BNAME" (B for both) with the idea of using it for completely aliasing a DNS subtree to cope with internationalized domain names that are semantically equivalent but have different Unicode encodings (e.g. ss / ß). However the records have to be put in the parent zone, which is problematic if the parent is a TLD.

The questions afterwards predictably veered towards ANAME and I spoke up to encourage the audience to take a look at my revamped ANAME draft when it is submitted. (I hope to do a submission early this week to give it a wider audience for comments before a revised submission near the deadline next Monday.)

Tale Lawrence mentioned the various proposals for multiple queries in a single DNS request as another angle for improving performance. (A super simplified version of this is actually a stealth feature of the ANAME draft, but don't tell anyone.)

I spoke to a few people about ANAME today and there's more enthusiasm than I feared, though it tends to be pretty guarded. So I think the draft's success really depends on getting the semantics right.

C-DNS / dnstap

Early in the morning was Jim Hague also of Sinodun talked about C-DNS, which is a compressed DNS packet capture format used for DITL ("day in the life" or "dittle") data collection from ICANN L-root servers. (There was a special DITL collection for a couple of days around the DNSSEC key rollover this weekend).

C-DNS is based on CBOR which is a pretty nice IETF standard binary serialization format with a very JSON-like flavour.

Jim was talking partly about recent work on importing C-DNS data into the ClickHouse column-oriented SQLish time-series database.

I'm vaguely interested in this area because various people have made casual requests for DNS telemetry from my servers. (None of them have followed through yet, so I don't do any query data collection at the moment.) I kind of hoped that dnstap would be a thing, but the casual requests for telemetry have been more interested in pcaps. Someone (I failed to make a note of who, drat) mentioned that there is a dnstap fanout/filter tool, which was on my todo list in case we ever needed to provide multiple feeds containing different data.

I spoke to Robert Edmonds (the dnstap developer, who is now at Fastly) who thinks in retrospect that protobufs was an unfortunate choice. I wonder if it would be a good idea to re-do dnstap using uncompressed C-DNS for framing, but I didn't manage to talk to Jim about this before he had to leave.

DNS Flag day

A couple of talks on what will happen next year after the open source DNS resolvers remove their workaround code for broken authoritative servers. Lots of people collaborating on this including Sebastián Castro (.nz), Hugo Salgado (.cl), Petr Špaček (.cz).

Their analysis is rapidly becoming more informative and actionable, which is great. They have a fairly short list of mass hosting providers that will be responsible for the vast majority of the potential breakage, if they aren't fixed in time.

Smaller notes

Giovane Moura (SIDN) - DNS Defenses During DDoS

• also to appear at SIGCOMM

• headline number on effectiveness of DNS caches: 70% hit rate

• query amplification during an outage can be 8x - unbound has mitigation for this which I should have a look at.

Duane Wessels (Verisign) - zone digests

• really good slide on channel vs data security

• he surprised me by saying there is no validation for zone transfer SOA queries - I feel I need to look at the code but I can imagine why it works that way...

• zone digests potentially great for safer stealth secondaries which we have a lot of in Cambridge

• Petr Spacek complained about the implementation complexity ... I wonder if there's a cunning qp hack to make it easier :-)

Peter van Dijk (PowerDNS) - NSEC aggressive use and TTLs

• there are now three instead of two TTLs that affect negative cacheing: SOA TTL, SOA MINIMUM, plus now NSEC TTL.

• new operational advice: be careful to make NSEC TTL and SOA negative TTLs match!