Tony Finch's blog

https://dotat.at/@/2022-12-04-leap-seconds.html

It has been a couple of years since my previous blog post about leap seconds, though I have been tweeting on the topic fairly frequently: see my page on date, time, and leap seconds for an index of threads. But Twitter now seems a lot less likely to stick around, so I'll aim to collect more of my thinking-out-loud here on my blog.

( Read more... )

Since my blog post about hg64 histograms in July I have continued occasional hacking on the code, so it's time to post an update.

As before, my proof-of-concept histogram data structure hg64 is available from:

• https://dotat.at/cgi/git/hg64.git
• https://github.com/fanf2/hg64

( Read more... )

https://dotat.at/@/2022-07-15-histogram.html

I have reached the point in my main project (qp-trie for BIND) where I want to collect performance statistics. Things like:

• how long it takes to compact a trie;
• how much memory was recovered by compaction;
• size of trie before and after compaction.

In a wider context, those using BIND for large authoritative server systems would like better statistics on things like:

• incoming and outgoing zone transfer times;
• zone refresh query latency.

For my own purposes, I'm no longer satisfied with summarizing performance with just the mean and standard deviation; and in an operational context, it's useful to know about the existence of outliers (rare outliers can be hidden by simple summary statistics) and the size of an outlier can be a useful clue.

So, we need histograms!

hg64

I have written a proof-of-concept histogram data structure called hg64, which you can get from:

• https://dotat.at/cgi/git/hg64.git
• https://github.com/fanf2/hg64

It can load 1 million data items in about 5ms (5ns per item), and uses a few KiB of memory.

Here I will write about how hg64 came to be.

( Read more... )

https://dotat.at/@/2022-06-27-tolower-swar.html

Here's a fun bit of Monday optimization.

DNS servers often have to convert DNS names to their canonical lower case form. BIND has to do so a bit more than most because it tries to preserve the case of DNS names rather than canonicalizing them.

I was recently distracted by a cleanup oportunity: there were multiple maptolower lookup tables in BIND, so I decided to make a home for them so that we would only need one copy. Then I thought, surely there are faster ways to implement tolower() than a lookup table.

( Read more... )

https://dotat.at/@/2022-06-04-metric.html

My response to the UK government consultation on using imperial units of measure for retail.

Regarding the foreword to the consultation description, my children were weighed in kilos at birth, so whoever wrote the foreword must be wery old, and talking about babies who are now grown adults.

And for metrological purposes, British weights and measures have been derived from the metric system since the international convention on the metre in 1875 and the production of the new metric prototypes in London in 1898, because the metric standards were made to a higher quality with greater reliability than the old imperial standards. So it's incorrect to say that we started adopting the metric system in 1995; that was actually when we started abolishing imperial (and that was at the initiative of the UK government, not the EU).

oops, I should have mentioned that the tories were in power in 1995

Q. 1 for all,

1 a) Are there any specific areas of consumer transactions that should be a priority for allowing a choice in units of measurement, and why?

None, because there is no need to make it more confusing to compare sizes and prices.

1 b) Are there any specific areas that you think should be excluded from a choice in units of measurement, and why?

There should be a single set of standard units of measure so that it is as easy as possible to compare the sizes and prices of goods, especially (e.g.) price per kilo or price per litre.

1 c) If an item is sold in imperial measures, should there be a requirement for a metric equivalent alongside it?

Metric measures should always be required, and there should always be clear labelling showing the price per some power of 10 (e.g. 100g or 1kg)

Q. 2 for businesses,

n/a (I am replying as an individual)

Q. 3 for consumers,

3 a) If you had a choice, would you want to purchase items: * (i) in imperial units? * (ii) in imperial units alongside a metric equivalent?

I always want to purchase items in metric (preferably a round number when measured in metric units), and I want it to be easy to find the quantity labelled in metric.

3 b) Are you more likely to shop from businesses that sell in imperial units?

I am likely to avoid shops that use imperial units, because the measurements will be confusing and unfamiliar.

3 c) Do you foresee any costs or benefits to you from businesses being permitted to sell: * (i) solely in imperial units? * (ii) in imperial units alongside a less prominent metric equivalent?

I expect it will be detrimental to allow businesses to sell in imperial units only, because it will make it easier for them to confuse customers, misprice goods, and hide their malpractice.

3 d) Do you have experience of buying solely in imperial units?

I am 47 years old and I cannot remember when imperial measures were in wide use for anything except milk and beer.

Milk, for example, is often confusing when 2 litre bottles are sold alongside 4 pint bottles, and I have to carefully examine the labels to find out which is the larger volume and what is the price per litre of the 4 pint bottle.

For beer, how can I compare a pint bottle and a 33cl bottle? It’s relatively easy for 50cl vs 33cl because the ratio is 3/2. In practice I think of a pint as about half a litre, so I can compare in metric. And, when I am served beer in a pub in a pint-to-brim glass, what I usually get is closer to half a litre than a pint.

Q. 4 for trading standards,

n/a (I am replying as an individual)

This week I am at an IETF meeting in Vienna. Yesterday afternoon I took a few pictures while I walked around the city; this blog item is about my non-touristy activity.

isc mfld

I started work for isc.org at the beginning of this month. We dealt with a lot of the new employee logistics before my official start date, like getting a new work laptop and sorting out logins for email and other accounts. And Jeff Osborne (president of ISC) encouraged me to come to come to IETF113 in Vienna to meet him and some of the other ISC staff in person soon after my start.

And so, the MFLD track (many fine lunches and dinners) has mostly been chatting with new colleagues.

qp-trie

One of our aims with BIND is to revamp its more difficult parts, trying to make them simpler, and maybe also improving performance. I come to ISC with my qp-trie data structure, which I hope can replace BIND's rbtdb, aka red-black tree database, BIND's in-memory data structure for looking up DNS records.

Last summer I did some work on NSD to see how my DNS-trie variant would work in the context of a real (but relatively simple) DNS server, and further experiments to try out a new copy-on-write concurrency strategy. During the IETF hackathon on Saturday and Sunday I revisited my NSD fork to remind myself which branch does what. I discussed it with Benno Overeinder and Willem Toorop from NLnet Labs, and wrote up some notes for them and myself. They are planning to revisit NSD's zone lookup data structures, and my qp-trie is one of the candidates.

bind

I had a discussion with my new colleague Petr Špaček about testing BIND's rbtdb refactoring, especially how to exercise things like locking that unit tests would not, and where we need better performance metrics. It's early days yet, but it will be helpful to get benchmark baselines in place soon.

radix trees

Petr introduced me to Jan Včelák; they both worked at cz.nic in the past, and know of my qp-trie work in Knot DNS. We had a chat about fast lookup for IP address prefixes, and Jan pointed me to a paper describing a "Tree Bitmap" that he had implemented.

There has been a lot of research into fast IP address prefix lookup, because it is in the fast path of network routers, and they often use the same popcount packed array trick as my qp-trie. There is a radix tree in BIND which is used for access control list matching, and which could definitely be made a lot faster.

dnsop

On Tuesday morning was the DNSOP working group meeting. I have not been paying much attention to DNSOP in the last couple of years, so my aim was mostly to get an idea of the current state of play. It turns out that the working group has been clearing its backlog, and is now considering new work for adoption. I came away with a couple of drafts that I should look at more closely.

First, the "DNS referral glue requirements" draft, where there are some questions on terminology that I have opinions about.

And second, "using service bindings with DANE" which gets into awkward questions about the semantics of aliases in the DNS, with particular reference to DANE SRV RFC 7673 which has my name at the top. I am not filled with joy at the prospect of arguing about this again, but I feel I ought to...

I have made a new release of nsnotifyd a tiny DNS server that just listens for NOTIFY messages and runs a script when one of your zones changes.

This nsnotifyd-2.0 release adds support for TCP. The original nsnotifyd only supported UDP, which works fine with BIND, but Knot DNS sends NOTIFY messages over TCP.

As well as its new TCP support, the nsnotify client program that sends NOTIFY messages can now send bulk notifications for lots of zones, as well as being able to send notifications to lots of recipient servers.

Many thanks to Niels Haarbo and DK Hostmaster for requesting TCP support and for sponsoring my work to implement it.

I like nsnotifyd because I wrote it in 2015, and I haven’t touched it since then (until this month). I usually hear nothing about nsnotifyd, but occasionally someone mentions they are using it. For example the Guardian tech blog said of nsnotifyd, "like all good *nix tools it does one thing well", and JP Mens called it "a gem of a utility".

Happy users, no bug reports, software bliss.

I have sent my formal resignation letter: I am leaving the University. From March, I will be working full-time for isc.org on BIND.

I said in my letter that it has been a privilege and a pleasure working for the University. That isn't a polite fiction; I like the people I have worked with, and I have had interesting and rewarding things to do. But this is an exciting opportunity which came at the right time, so I allowed myself to be dragged away.

I have been a contributor to BIND for several years now. When I met Jeff Osborne (the president of ISC) in Amsterdam 3 years ago, he joked that I worked for them but he didn't pay me :-) And I have met many of the ISC folks at various conferences. So in some ways it's known territory.

But some things are going to be very different. The university has more than 1000 times the number of people as ISC, whereas ISC is almost entirely remote so spans over 1000 times the area. From my own point of view, I am looking forward to working in a bigger team than I have been.

This all happened fairly fast: Ondřej Surý (head of BIND development) suggested the new job to me only a few weeks ago, and I still find it hard to believe it's really happening! We have talked over email about what Ondřej would like me to work on, and I'm looking forward to getting stuck in.

Here are some ideas for how a special-purpose allocator might improve a qp-trie implementation:

• lower memory usage
• faster allocation
• neater RCU support
• possibly less load on the TLB
• tunable fragmentation overhead

The downsides are:

• complexity - it's a custom allocator and garbage collector!
• it would only support transactional updates

Let's dig in...

COW and RCU

A few years ago I added support for transactional updates to the qp-trie used by Knot DNS. Knot handled DNS updates by making a complete copy of the zone, so that the old copy could continue to serve queries while the new copy was being modified. The new code made it possible to copy only the parts of the zone that were affected by the update, and reduce the overhead of handling small updates.

My COW (copy-on-write) code was designed to work with the RCU (read-copy-update) concurrency framework. RCU was developed for concurrent data structures in the Linux kernel; there is also a userland RCU library. RCU is a combination of;

• COW data structures, so updates don't interfere with readers
• lightweight concurrency barriers, so readers do not need to take a lock
• deferred cleanup, so writers know when all readers are using the new copy of a data structure, when the old copy can be cleaned up

I used one-bit reference counts to mark the boundary between the parts of the tree (mostly near the root) that had been copied, and the shared parts (towards the leaves). So it wasn't a pure COW, because the refcount manipulation required writes to the shared parts of the tree.

memory layout

A common design for malloc() implementations (for example phkmalloc and jemalloc) is to keep allocations of different sizes separate. Each size class has its own free list, and each page can only satisfy allocations from a single size class. This can reduce the amount of searching around for free space inside malloc() and reduce the amount of fragmentation.

But in a qp-trie, nodes are often different sizes, so each step when traversing the tree will usually require a leap to a different page, which can increase pressure on the CPU's address translation lookaside buffer.

Could we, perhaps, make a qp-trie more friendly to the TLB, and maybe also the prefetcher, by being more clever about how it allocates nodes, and how they are arranged next to each other in memory?

A custom allocator seems like a lot of work for a (probably) small performance improvement, so I have not (until recently) pursued the idea.

refcounts vs tracing

Reference counts are often regarded as a poor substitute for "proper" tracing garbage collection. A tracing copying collector can give you:

• cheaper allocations: just bump a pointer
• amortized free: release whole pages rather than individual nodes
• better locality and less fragmentation
• no extra write traffic to update reference counts

To get most of these advantages, the garbage collector must be able to move objects around. What you gain in more efficient alloc and free, you pay for by copying.

However, if all updates to our data structure are RCU transactions that necessarily involve making copies, then tracing garbage collection seems like less of a stretch.

rough design

Our qp-trie allocator has a bag of pages, whose size does not need to match the hardware page size, but that kind of size should be about right.

For each page, we keep track of how much free space it has (so that we can decide when it is worth evacuating and freeing it), and a note of the RCU epoch after which it can be freed.

There's a global array of pages, containing the the address of each page. Actually, when the page table is resized, we will need to do an RCU delayed cleanup, so there can also be a secondary array which is waiting to be freed.

starting a transaction

When an update transaction is started, we obtain a fresh page where we will put new nodes and modified copies. We use a cheap bump allocator that just obtains another page when it runs out of space. Unlike many GC languages, we still manually free() nodes, to keep count of the free space in each page.

There can only be one write transaction at a time, so the writer can update the page metadata without interlocks.

finishing a transaction

After the updates have been applied to make a new version of the tree, we can do a bit of extra maintenance work before switching our readers over to the new tree. I'll discuss these in more detail below:

• layout optimization: it might be worth doing some extra copying to make the tree nicer for the prefetcher;

• garbage collection: identify which pages have too much free space, and evacuate and compact their contents so they can be freed.

• cache eviction: if our tree is used for a cache rather than for authoritative data, the GC phase can also discard entries that are past their TTL.

Finally, the switch-over process:

• swap the tree's root pointers atomically
• wait for an RCU epoch so all readers are using the new tree
• free everything on the delayed cleanup list

layout optimization

This is entirely optional: I don't know if it will have any useful effect. The idea is to copy tree nodes into a layout that's friendly to the CPU's prefetcher and maybe also its TLB. My best guess for how to achieve this is, starting from the root of the tree, to copy nodes in breadth-first order, until some heuristic limits are reached.

One of the tradeoffs is between better layout and extra memory usage (for more copies). A minimal option might be to only copy the few uppermost levels of the tree until they fill one page. Layout optimization across multiple pages is more complicated.

garbage collection

Here is a sketch of an algorithm for a full collection; I have not worked out how to do a useful collection that touches less data.

We recursively traverse the whole tree. The argument for the recursive function is a branch twig, i.e. a pointer to an interior node with its metadata (bitmap etc.), and the return value is either the same as the argument, or an altered version pointing to the node's new location.

The function makes a temporary copy of its node on the stack, then iterates over the twigs contained in the node. Leaf twigs are copied as is; it calls itself recursively for each branch twig.

If any of the branch twigs were changed by the recursive calls, or if the old copy of this node was in a sufficiently-empty page, the old copy is freed (which only alters its page's free space counter), the new version of the node is copied to the allocation pointer, and this recursive invocation returns the node's new location. Otherwise it returns a pointer to the old location (and the copy on the stack is discareded).

We can tune our fragmentation overhead by adjusting the threshold for sufficiently-empty pages. Note that garbage collection must also include recent allocations during the update transaction: a transaction containing multiple updates is likely to generate garbage because many qp-trie updates change the size of a node, even if we update in place when we can. So the pages used for new allocations should be treated as sufficiently-empty so that their contents are compacted before they enter heavy read-only use.

cache eviction

So far my qp-trie code has worked well for authoritative data, but I have not tried to make it work for a DNS cache. A cache needs to do a couple of extra things:

• evict entries that have passed their time-to-live;
• evict older entries to keep within a size limit.

Both of these can be done as part of the garbage collection tree walk.

In BIND, activities like this are performed incrementally by co-operatively scheduled tasks, rather than dedicated threads, which makes them a bit more intricate to code.

small pointers

The page table allows us to use much smaller node pointers.

Instead of using a native 64-bit pointer, we can refer to a node by the index of its page in the page table and the position of the node in its page, which together can easily fit in 32 bits. This requires a double indirection to step from one node to the next, but the page table should be in cache, and qp-trie traversal is friendly to prefetching, so we can provide hints if the processor can't prefetch automatically.

There are a couple of ways to make use of this saving.

We can reduce the size of each twig from 16 bytes to 12 bytes, making the whole tree 25% smaller. This adds some constraints on leaves: either the key and value pointers must fit in 48 bits each (which requires unwarranted chumminess with the implementation); or we can get hold of the key via the value (and waste 32 bits in the leaf).

Or if this is a one-pass DNS-trie we can use the extra space for path compression, and avoid making assumptions about pointers.

metadata placement

For each page we need to keep track of how much free space it contains, so that we know when it should be evacuated; and something to tell us if the page should be freed after the next RCU epoch.

It's fairly straightforward to put this metadata at the start of each page. At the cost of a little wasted space we can make sure this writable data doesn't share a cache line with read-only nodes.

If we are using small pointers, another option is to put per-page metadata in the page table, or perhaps in another array parallel to the page table to keep read-only and writable data separate.

transactions and caches

I normally think of a cache as having a lot of small point updates, which is unlikely to be efficient with this transaction-oriented design. But perhaps it makes sense if we split the cache into two parts.

The main cache is read-only; we use transactional updates for eviction based on TTL and cache size, and to bring in new records from the working cache. It uses ordered lookups to support RFC 8198 NXDOMAIN synthesis.

The working cache is used by the resolver to keep track of queries in progress. It can be based on fine-grained updates and locking, rather than being designed for a read-mostly workload. It might not need ordered lookups at all.

Queries that miss the main cache get handed over to the resolver, which might be able to answer them straight from the working cache, or it might add the query to a list of queries waiting for the same answer, or when there are no matches it creates a new entry in the working cache that belongs to a new resolver task.

application data

Most of what I have written above is about working with the interior branch nodes of the tree. What about the application data hanging off the leaf nodes?

During a transaction, any elements that we want to delete need to be added to a free list, so that they can be cleaned up after the next RCU epoch. When we need to modify an element, we must do so COW-style.

It's reasonable for the tree implementation to keep a free list of application elements, so any delete or set operations will automatically add the old element pointer to the list for later cleanup. On the other hand, it's probably easier for the applicaton to COW its own data.

The only callback we will need is to free application elements during the delayed cleanup after the RCU epoch has passed. (This is simpler than Knot DNS, which also has callbacks for refcount manipulation.)

conclusion

For a long time I was doubtful that a custom allocator for a qp-trie would be worth the effort. But now I think it is likely to be worth it:

• The refcounting in Knot is confusing; GC seems to be a nicer way to support RCU.

• Small pointers can save a significant amount of space, and are more accommodating for a one-pass radix tree version.

• It will become feasible to see if layout optimization can make queries faster.

It remains to be seen if I can find the time to turn these ideas into code!

Nautical miles and metres were both originally defined in the same way: each of them was the distance on the surface of the earth subtended by a particular angle. For nautical miles, the angle is a minute of arc; for the metre, it is 1/40,000,000 of a turn.

I was idly wondering about the history of this way of defining a unit of length, and how it led to these two particular units, and what (if any) influence they had on each other. The In Our Time episode on Pierre-Simon Laplace briefly discussed his involvement in the definition of the metric system in revolutionary France, which nudged me to actually do some reading.

( Read more... )

Last summer I wrote about my DNS-trie, a version of my qp-trie that is optimized for DNS names. It turned out to work rather well: I patched NSD to use my code and it was smaller and faster than NSD's radix tree.

But there seemed to be a couple of places where there was room for improvement.

( Read more... )

One of the small annoyances about Mac OS Terminal.app is that is claims to be xterm-256color but its default key escape codes are incomplete and somewhat weird. (For example, alt-left generates ESC-b which is the Emacs backward-word binding, rather than a VT-style escape.)

I wanted to fix this properly, so here's what I did.

( Read more... )

These are some notes on what I have done with Rust over the last week, in the course of doing something highly experimental that could well turn out to be foolishly over-complicated. (I'll be interested to hear about other ways of doing this, in other programming languages as well as Rust!) But it involves some cool ideas - abstract interpretation, type-level programming in Rust, automatically optimized data structures, and a bit of geometric algebra for 3D graphics - and I want to share what I learned.

Health warning: this did turn out to be foolishly over-complicated - I suggest reading the followup before wading in to the rest of these notes, if you have not already read them.

( Read more... )

I recently saw Colm MacCárthaigh's exegesis of Daniel Lemire's paper about nearly divisionless fast random integer generation. I found myself writing down my own notes on how it works, that take a slightly different angle.

( Read more... )

In 2010 I wrote down my idea for generalized string literals.

A lot has changed in programming languages since then: Go, Kotlin, Rust, and Swift have become important and influential; C++ and Javascript have been significantly revamped. String literals in many languages are a lot more complicated than they were 10 years ago.

My design has evolved a little bit since my old description, and recently it has been agitating me for another write-up and a comparison with non-fantasy literal syntaxes.

( Read more... )

What is the right way to understand endianness?

In my view endianness makes much more sense when viewed as a matter of bit order: these notes explore how byte order and byte swapping are a consequence of bitwise endianness combined with bytewise addressing and data transmission. But byte swapping isn't the only flavour of endianness: bitwise endianness is equally comfortable with oddities like PDP11 word swapping.

I also explain why big-endian is just as mathematically elegant as little-endian: they are perfectly dual to each other, so one can't be more elegant than the other.

There's a sketch of an abstract API for endian-aware bit sequences, designed with algebra in mind (associativity, anamorphisms and catamorphisms).

edited to add: this article has a sequel with improved symmetry

( Read more... )

I'm sitting in my room surrounded by un-assembled bits of IKEA furniture and un-opened boxes containing various goodies to help organize my clutter. I have been distracted by an idea that would not let go. It's not often I get literally compelled by creativity!

This is the story of my splendid new DNS-trie code, with benchmarks at the end. ( It is quite long so it is behind a cut... )

For several years I have published a leap second table in the DNS. My version(s) of the leap second table are (as far as I know) unique:

• They are cryptographically signed: if you get my leap second table you know it came from my workstation.

• They have the full history from 1972, in a single DNS query, and this needs less than 100 bytes (plain text) or 20 bytes (binary).

My leap second table is sod-all use

The problem with leap seconds is not how they are published, but the tools that (fail to) disseminate and use the leap second table.

NTP is a big problem: the traditional NTP implementation is designed to choose and fixate on one server out of an ensemble of possible upstreams. Leap seconds are not part of the decision process of choosing which upstream is correct.

As a result, NTP will trust an upstream that is good at keeping time (maybe it is in a room with good temperature stability), but misconfigured for leap seconds, so when a leap second comes along NTP gets surprised by a sudden one-second offset.

Leap seconds should not be a surprise

There at least two plausible ways to avoid being surprised:

• Make your NTP server get the leap second table from a trusted source, e.g. my DNS records. If your upstream NTP server's leap indicator bits disagree with the correct table, expect that it will go wrong when a leap second happens.

• Get NTP servers to disseminate the table, and only trust upstreams that agree with the majority. The table is less than 20 bytes (600x smaller than the NIST table) so it can easily fit in an NTP packet.

How to fix leap seconds

• Make it cheap to distribute the leap second table.

  Done, but my spec needs more rigor. My binary leap second table is smaller than a SHA-1 hash, so it is cheaper to distribute the whole thing than a digest.

• Make NTP distrust upstreams that disagree with the leap second table when a leap second is pending.

  This is the hard part, because it involves persuading multiple open source and proprietary implementers.

I am not working on this

But I will happily rant about precise time keeping at the Cambridge Beer Festival to anyone who talks about their project to refurbish a rubidium frequency standard [not a hypothetical example] and I will try not to bore people who are not time nuts....

I thought that this week I might work on polishing my spec for compact leap second tables, or writing an implementation in Rust, but I have decided that tackling the oppressive todo list of procrastinatory doom will make me happier.

I have had this article brewing for some time now but it has never really had a point. I think the drafts in my head were trying to be too measured, even handed, educational, whereas this probably works better if I be an old man shaking my fist at a cloud.

What the fuck is a sysadmin anyway

Go and read Rachel Kroll on being a sysadmin or what.

Similar to Rachel, when I was starting my career I looked up to people who ran large systems and called themselves sysadmins: at that time the bleeding edge of scalability was the middle of the sigmoid adoption curve in universities, and in early ISPs, so "large" was 10k - 100k users. And these sysadmins were comfortable with custom kernels and patched daemons.

The first big open source project I got involved with was the Apache httpd, which was started by webmasters who had to fix their web servers, and who helped each other to solve their problems. Hacking C to build the world-wide web.

About ten years later, along came DevOps and SRE, and I thought, yeah, I code and do ops, so what? I like the professionalism both of them have promoted, but they tend to be about how to run LOTS of bespoke code.

Are you local?

Go and read David MacIver on "situated code".

A lot of the code I have inherited and perpetrated has been glue code that's inherently "situated" - tied to a particular place or context. ETL scripts, account provisioning, for example.

There are actually two dimensions here: situatedness (local vs. global) and configness vs hardcodedness. The line between code and config is blurry: If you can't configure a feature, do you write a wrapper script, or do you hack the code to add it?

Local patches are just advanced build-time configuration.

Docker images are radically less-situated configuration.

Custom code is bureaucratic overhead

Code is not as bad as personal data - that's toxic waste. Code is more like a costly byproduct of providing a service. Write code to reduce the drudgery of operating the service; then operations becomes maintaining the bespoke code.

Maintaining the code becomes bureaucratic overhead. A bureaucracy exists to sustain itself.

How to reduce the overhead? DELETE THE CODE. How do you do that? Simplify the code. Share the code. Offload the code.

There is no open source business model

Unless you are Red Hat IBM.

There has been a lot of argument in recent months about open source companies finding it hard to make money when all the money is going to AWS.

Go and read this list of fail.

The code I write is a by-product of providing a service. This is how Apache httpd and Exim came to be. The point was not to make money from the software, the point was to make some non-software thing better. And sharing improvements to code that solves common problems is the point of open source software.

Doing open source wrong

Don't solve a problem with open source software using code that you can't share.

Amazon has an exceptionally strict policy of taking open source code and never sharing any of the improvements they make. Their monopolizing success is the main cause of the recent crisis amongst open source software businesses. It isn't open source's fault, it's because Amazon are rapacious fuckers, and monopolies have somehow become OK.

Doing open source right

Everything I have learned about software quality in practice I have learned from open source.

From the ops perspective, before you can even start to consider the usual measures of quality (documentation, testing, reliability ...) open source forces you to eliminate situatedness. Make the code useful to people other than yourselves. Then you can share it, and if you are lucky, offload it.

If some problem is difficult to solve with your chosen package, you can often solve it with a wrapper script or dockerfile. You can share your solution in a blog post or on GitHub. That's all good.

Even better if you can improve the underlying software to make the problem easier to solve, so the blog posts and wrapper scripts can be deleted. It's a lot more work, but it's a lot more rewarding.

This was a good way to eat up the left-over veg from xmas day.

Yesterday we had chicken with roast potatoes, leek sauce, peas, and pigs in blankets - not a complicated meal because there was just the four of us.

I cooked a 1kg bag of roasties, which we ate a bit more than half of, and I made leek sauce with two leeks of which about half was left over, plus about 100g of left-over peas. Obviously something like bubble and squeak or veggie potato fritter things is the way to use them up.

Here are some outline recipes. No quantities, I'm afraid, because I am not a precision cook.

Leek sauce

I dunno how well-known this is, but it's popular in my family.

Slice a couple of leeks into roundels, and gently fry them in LOTS of butter until they are soft.

Then make a white sauce incorporating the leeks. So add a some flour, and stir it all together making a leeky roux.

Then add milk a glug at a time, stirring to incorporate in between (it's much more forgiving than a plain white sauce) until it is runny. Season with nutmeg and black pepper.

Keep stirring while it cooks and the sauce thickens.

Ham

I like to cook a ham like this on Boxing Day.

Get a nice big gammon, which will fit in one of your pots while covered in liquid. It's best to soak it in cold water in the fridge overnight, to make it less salty.

Drain it and rinse it, then put it on the hob to simmer in cider (diluted to taste / budget with apple juice and / or water). I added some peppercorns and bay leaves, but there are lots of spice options.

For the glaze I mix the most fierce mustard I have with honey, and (after peeling off the skin) paint it all over the gammon. Then roast in an oven long enough to caramelize the glaze (15 minutes ish).

Let the ham stand - it doesn't need to be hot when you serve it.

Cider gravy

In the past I have usually neglected to soak the ham before cooking, so the cider ended up impossibly salty afterwards. This time I tried reducing the cooking liquid to see if I could make a cider gravy, but it was still too salty.

Apparently I could try making a potato soup with it because that neutralizes the salt. I think I will try this because we also have loads of (unsalted) chicken stock to be used.

Bubble and squeak fritters

This might have worked better if I had got the leftovers out of the fridge some time before preparation, so that they could warm up and soften!

I mashed the potatoes, then mixed them together with the leek sauce (which I zapped in the microwave to make it less solid!), and finally added the peas (to avoid smashing them up too much). I also added an egg, but actually the mixture had enough liquid already and the egg made it a bit too soft.

After some experimentation I found that the best way to cook it was to put a dollop of mixture directly in the frying pan with a wooden spoon, so (after flattening) the fritters were about 2cm thick and a bit smaller than palm sized. I could comfortably cook a few at a time.

I fried them in oil in a moderately hot pan, so they were hot all the way through and browned nicely on the outside.

Verdict

I had just enough fritters to feed three adults and a child, and most of the ham has gone! Enough left for a few sandwiches, I think.

I will try to aim for left-over leek sauce and potatoes more often :-)

(Fri Sat Sun Mon Tue Wed Thu)

Very belated, but there was not a great deal to report from the last day of the RIPE meeting and I have spent the last few days doing Other Things.

One of the more useful acronyms I learned was the secretly humorous "DONUTS": DNS Over Normal Unencrypted TCP Sessions.

Following Thurday's presentation by Jen Linkova on IETF IPv6 activity, Sander Steffann did a lightning talk about the IPv6-only RA flag. There was quite a lot of discussion, and it was generally agreed to be a terrible idea: instead, operators who want to suppress IPv4 should use packet filters on switches rather than adding denial-of-service features to end hosts.

Amanda Gowland gave a well-received talk on the women's lunch and diversity efforts in general. There was lots of friendly amusement about there being no acronyms (except for "RIPE").

Razvan Oprea gave the RIPE NCC tech report on the meeting's infrastructure:

• 10Gbit connection to the meeting - "you haven't used much of it to be honest, so you need to try harder" - peaking at 300Mbit/s

• 800 simultaneous devices on the wireless net

• They need more feedback on how well the NAT64 network works

• There were a few devices using DNS-over-TLS on the Knot resolvers

One of the unusual and popular features of RIPE meetings is the real-time captioning produced by a small team of stenographers. In addition to their normal dictionary of 65,000 common English words, they have a custom dictionary of 36,000 specialized technical terms and acronyms. Towards the end of the week they relaxed a bit and in the more informal parts of the meeting (especially when they were being praised) they talked back via the steno transcript display :-) (Tho those parts aren't included in the steno copy on the web).

That's about it for the daily-ish notes. Now to distill them into an overall summary of the week for my colleagues...

I was out late last night so I'm writing yesterday's notes this morning.

Yesterday I attended the DNS and MAT meetings, and did some work outside the meetings.

CDS

Ondřej Caletka presented his work on keeping DNS zone files in git.

• Lots of my favourite tools :-) Beamer, Gitolite, named-compilezone

• How to discover someone has already written a program you are working on: search for a name for your project :-)

BCP 20 classless in-addr.arpa delegation led to problems for Ondřej: RFC2317 suggests putting slashes in zone names, which causes problems for tools that want to use zone names for file names. In my expired RFC2317bis draft I wanted to change the recommendation to use dash ranges instead, which better matches BIND's $GENERATE directive.

At the end of his talk, Ondřej mentioned his woork on automatically updating the RIPE database using CDS records. As planned, I commented afterwards in support, and afterwards I sent a message to the dns-wg mailing list about CDS to get the formal process moving.

DNS tooling

I spoke to Florian Streibelt who did the talk on BGP community leaks on Tuesday. I mentioned my DNS-over-TLS measurements; he suggested looking for an uptick after christmas, and that we might be able to observe some interesting correlations with MAC address data, e.g. identifying manufacturer and age using the first 4 octets of the MAC addresss. It's probably possible to get some interesting results without being intrusive.

I spent some time with Jerry Lundstrom and Petr Špaček to have a go at getting respdiff working, with a view to automated smoke testing during upgrades, but I ran out of battery :-) Jerry and Petr talked about improving its performance: the current code relies on multiple python processes for concurrency.

I talked to them about whether to replace the doh101 DNS message parser (because deleting code is good): dnsjit message parsing code is C so it will require dynamic linking into nginx, so it might not actually simplify things enough to be worth it.

DNS miscellanea

Ed Lewis (ICANN) on the DNSSEC root key rollover

• next step is January 11 when the old key gets revoked and the DNSKEY response size will grow a few bytes bigger than it has been before

• Geoff Huston says see http://www.potaroo.net/ispcol/2017-08/xtn-hdrs.html and http://www.potaroo.net/ispcol/2016-11/rootstars.html for information about the risks of the large response size

Petr Špaček (CZ.NIC) on the EDNS flag day, again

• "20 years is enough time for an upgrade"

Ermias Malelgne - performance of flows in cellular networks

• DNS: 2% lookups fail, 15% experience loss - apalling!

Tim Wattenberg - global DNS propagation times

• zero TTLs actually work! (with caveats!)

• https://ismydnslive.com/ propagation checker using RIPE Atlas

Other talks

Maxime Mouchet - learning network states from RTT

• traceroute doesn't explain some of the changes in delay

• nice and clever analysis

Trinh Viet Doan - tracing the path to YouTube: how do v4 and v6 differ?

• many differences seem to be due to failure to dual-stack CDN caches in ISP networks

Kevin Vermeulen - multilevel MDA-lite Paris traceroute

• MDA = multipath detection algorithm

• I need to read up on what Paris traceroute is ...

• some informative notes on difficulties of measuring using RIPE Atlas due to NATs messing with the probe packets

Same city, same hotel, same lunch menu, but we have switched from DNS-OARC (Fri Sat Sun) to RIPE, which entails a huge expansion in the bredth of topics and number of people. The DNS-OARC meeting was the biggest ever with 197 attendees; the RIPE meeting has 881 registrations and at the time of the opening plenary there were 514 present. And 286 first-timers, including me!

I have some idea of the technical side of RIPE meetings because I have looked at slides and other material during previous meetings - lots of great stuff! But being here in person it is striking how much of an emphasis there is on social networking as well as IP networking: getting to know other people doing similar things in other companies in other countries seems to be a really important part of the meeting.

I have met several people today who I only know from mailing lists and Twitter, and they keep saying super nice things :-)

I'm not going to deep link to each presentation below - look in the RIPE77 meeting programme for the presentation materials.

DoT / DoH / DoQ

The DNS continues to be an major topic :-)

Sara Dickinson did her "DNS, Jim, but not as we know it" talk to great acclaim, and widespread consternation.

Ólafur Guðmundsson did a different talk, called "DNS over anything but UDP". His main point is that DNS implementations have appallingly bad transport protocol engineering, compared to TCP or QUIC. This affects things like recovery from packet loss, path MTU discovery, backpressure, and many other things. He argues that the DNS should make use of all the stateful protocol performance and scalability engineering that has been driven by the Web.

Some more or less paraphrased quotes:

• "I used to be a UDP bigot - DNS would only ever be over UDP - I was wrong"

• "DoH is for reformed script kiddies who have become application developers"

• "authenticated connections are the only defence against route hijacks"

• "is the community ready if we start moving 50% - 60% of DNS traffic over to TCP?"

I've submitted my lightning talk again, though judging from this afternoon's talks it is perhaps a bit too brief for the RIPE 10 minutes lightning talk length.

ANAME

Mattijs Mekking read through the ANAME draft and came back with lots of really helpful feedback, with plenty of good questions about things that are unclear or missing.

It might be worth finding some time tomorrow to hammer in some revisions...

Non-DNS things

First presentation was by Thomas Weible from Flexoptix on 400Gb/s fibre.

• lovely explanation of how eye diagrams show signal clarity! I did not previously understand them and it was delightful to learn!

• lots of details about transceiver form factors

• initial emphasis seems to be based on shorter distance limits, because that is cheaper

Steinhor Bjarnason from Arbor talked about defending against DDoS attacks.

• scary "carpet bombing", spreading DDoS traffic across many targets so bandwidth is low enough not to trigger alarms but high enough to cause problems and really hard to mitigate

• networks should rate-limit IP fragments, except for addresses running DNS resolvers [because DNS-over-UDP is terrible]

• recommended port-based rate-limiting config from Job Snijders and Jared Mauch

Hisham Ibrahim of RIPE NCC on IPv6 for mobile networks

• it seems there is a lot of confusion and lack of confidence about how to do IPv6 on mobile networks in Europe

• we are well behind the USA and India

• how to provide "best current operational practice" advice?

• what to do about vendors that lie about IPv6 support (no naming and shaming happened but it sounds like many of the people involved know who the miscreants are)

During IETF 101 I wrote up the day's activity on the way home on the train, which worked really well. My vague plan of doing the same in Amsterdam might be less successful because it's less easy to hammer out notes while walking across the city.

Some semi-chronological notes ... I'm going to use the workshop programme as a starting point, but I'm not going to write particularly about the contents of the talks (see the workshop web site for that) but more about what I found relevant to me and the follow-up hallway discussions.

Morning

The first half of the morning was CENTR members only, so I went to get a few bits from the local supermarket via the "I Amsterdam" sign before heading to the conference hotel.

The second half of the morning was DNS-OARC business. It's a small organization that provides technical infrastructure, so most of the presentation time was about technical matters. The last sessopn before lunch was Matt Pounsett talking about sysadmin matters. He's impressively enthusiastic about cleaning up a mountain of technical debt. We had a chat at the evening social and I tried to convince him that Debian without systemd is much less painful than the Devuan he has been struggling with.

Jerry Lundstrom talked about his software development projects. A couple of these might be more useful to me than I was previously aware:

• drool - DNS replay tool. I have a crappy dnsmirror script which I wrote after my previous test procedure failed to expose CVE-2018-5737 because I wasn't repeating queries.

  drool respdiff sounds like it might be a lot of what I need to automate my testing procedures between deploying to staging and production.

• dnsjit - suite of DNS-related tools in Lua. If this has the right facilities then I might be able to delete the horrid DNS packet parsing code from doh101.

Cloudflare DoH and DoT

Ólafur Guðmundsson said a few noteworthy things:

• Cloudflare's DoT implementation is built-in to Knot Resolver. They are heavy users of my qp trie data structure, and at the evening social Petr Špaček told me they are planning to merge the knot-reolver-specific fixes with my beer festival COW work.

• Their DoH implementation uses Lua scripting in NGINX which sounds eerily familiar :-) (Oversimplifying to the point of being wrong, Cloudflare's web front end is basically OpenResty.)

• He mentioned a problem with hitting quotas on the number of http2 channels opened by Firefox clients, which I need to double check.

• Cloudflare are actively working on DoT for recursive to authoritative servers. Sadly, although the IETF DNS Privacy working group has been discussing this recently, there hasn't been much comment from people who are doing practical work. Cloudflare likes the advantages of holding persistent connections from their big resolvers to popular authoritative servers, which is basically optimizing for a more centralized web. It's the modern version of leased lines between mainframes.

I submitted my DoH / DoT lightning talk (2 slides!) to the program committee since it includes stuff which Ólafur didn't mention that is relevant to other recursive server operators.

ANAME

I merged my revamped draft into Evan Hunt's aname repository and collaboration is happening. I've talked about it with at least one big DNS company who have old proprietary ANAME-like facilities, and they are keen to use standardization as a push towards removing unused features and cleaning up other technical debt. I've put some "mighty weasel words" in the draft, i.e. stealing the "as if" idea from C, with the idea that it gives implementers enough freedom to make meaningful zone file portability possible, provided the zone is only relying on some common subset of features.

Other small notes

Matt Larson (ICANN) - rollover was a "pleasant non-event"

• several ICANN staff are at DNS-OARC, so they travelled to NL before the rollover, and used a conference room at NLnet Labs as an ad-hoc mission control room for the rollover

Matt Weinberg (Verisign) - bad trust anchor telemetry signal investigation - culprit was Monero! I wonder if this was what caused Cambridge's similar weirdness back in June.

Duane Wessels (Verisign) talked about abolishing cross-TLD glue. Gradually the registry data model becomes less insane! [fanf's rant censored]

Jaromír Talíř (CZ.NIC) on TLD algorithm rollovers. Don't fear ECDSA P256! I definitely want to do this for Cambridge's zones.

Bert Hubert (PowerDNS) talked about the DNS Camel and his Hello DNS effort to make the RFCs easier to understand. He's very pleased with the name compression logic he worked out for tdns, a teaching DNS server. I want to have a proper look at it...

This afternoon I reckon I was six deep in a stack of yaks that I needed to shave to finish this job, and four of them turned up today. I feel like everything I try to do reveals some undiscovered problem that needs fixing...

• When the network is a bit broken, my DNS servers soon stop being able to provide answers, because the most popular sites insist on tiny TTLs so they can move fast and break things.

  As a result the DNS gets the blame for network problems, and helpdesk issues get misdirected, and confusion reigns.

• Serve-Stale to the rescue! It was implemented towards the end of last year in BIND and is a feature of the 9.12 releases.

  • Let's deploy it! First attempt in March with 9.12.1.

  • CVE-2018-5737 appears!

    Roll back!

  • The logging is too noisy for production so we need to wait for 9.12.2 which includes a separate logging category for serve-stale.

  • Time passes...

  • Deploy 9.12.2 earlier this week, more carefully.

  • Let's make sure everything is sorted before we turn on serve-stale again! (Now we get to today.)

    • The logging settings need revising: serve-stale is enough of a shove to make it worth reviewing other noisy log categories.

    • Can we leave most of them off most of the time, and use the default-debug category to let us turn them on when necessary?

    • This means the debug 1 level needs to be not completely appalling. Let's try it!

      • Hmm, this RPZ debug log looks a bit broken. Let's fix it!

      • Two little patches, one cosmetic, one a possible minor bug fix.

        • Need to rebase my hack branch onto master to test the patches.

        • Fix dratted merge conflicts.

      • Build patched server!

        • Build fails :-( why?

        • No enlightenment from commit logs.

        • Sigh, let's git bisect the build system to work out which commit broke things...

          • While the workstation churns away repeatedly building BIND, let's get coffee!

        • Success! The culprit is found!

        • Submit bug report

        • Work around bug, and get a successful build!

      • Test patched server!

        • The little patches seem OK, but while repeatedly restarting the server, a more worrying bug turns up!

          Sometimes when the server starts, my monitoring queries get stuck with SERVFAIL responses when they should succeed! Why?

        • Really don't want this to be anything that might affect production, so it needs investigation.

        • Turn off noisy background activity, and reproduce the problem with a simpler query stream. It's still hard to characterize the bug.

          • I'll need to test this in a less weird and more easily reconfigured server than my toy server. Let's spin up a VM.

            • Damnit, my virtualbox setup was broken by the jessie -> stretch upgrade!

            • Work out that this is because virtualbox is no longer included in stretch and the remnants from jessie are not compatible with the stretch kernel.

            • Reinstall virtualbox direct from Oracle. It now works again.

          • Install BIND on the new VM with a simplified version of my toy config. Reproduce the bug.

        • Is it related to serve-stale? no. QNAME minimization? no. RPZ? no.

        • After much headscratching and experimentation, enlightenment slowly, painfully dawns.

        • Submit bug report

          Actually, the writing of the bug report, and especially the testing of the unfounded assertions and guesses as I wrote it, was a key part of pinning down this weirdness.

          I think this is one of the most obscure DNS interoperability problems I have investigated!

OK, that's it for now. I still have two patches to submit, and a revised logging configuration to finalize, so I can put serve-stale into production, so I can make it easier in some situations for my colleagues to tell the difference between a network problem and a DNS problem.