More SMTP optimization

[fanf]

A couple of weeks ago I was working on the SMTP QUICKSTART specification, which optimizes the start of the SMTP connection.

http://www-uxsup.csx.cam.ac.uk/~fanf2/hermes/doc/qsmtp/draft-fanf-smtp-quickstart.html

I have now written a specification for SMTP transaction replay, which allows you to use the existing CHUNKING and PIPELINING extensions to acheive the maximum speed possible for bulk data transfer, whilst still behaving correctly if the connection is lost. I imagine this will be most useful clients on slow intermittent links, like mobile wireless users or people on the wrong end of a satellite link.

http://www-uxsup.csx.cam.ac.uk/~fanf2/hermes/doc/qsmtp/draft-fanf-smtp-replay.html

Gosh, over 5000 words...

Comments

[pm215]

Interesting. If you happen to have a cooperating client and server (as you might well in the mobile phone case, say) are you allowed to implement your own private SMTP extensions, or are you obliged to wait for them to become proper RFCs?

there's a typo (REPLAT) in s10.1, BTW...

[fanf]

Yes, private extensions are permitted. (Though you are supposed to use names starting woth X for non-standards-track extensions.)

[bens_dad]

> The server MUST require that the client uses the same security profile for a
> replayed transaction as it did for the original transaction. For example, if
> the client used TLS for the original transaction [RFC3207], it MUST also
> use TLS for the replayed transaction

I'm happy that this is the rule, to enforce security.

Will the client be allowed to switch between smtps and starttls,
or between ports (25, 587, 465), as long as they always use TLS ?
I'm thinking about the case where my mobile ISP happily allows port 25
but the connection dies so I retry over my home ISP which blocks port 25 ?

Are message-size values typically accurate to the octet ? I remember bugs in exim where it wasn't always correctly updated - I think for header-rewrites. Clearly this is a problem with the submitting application, but if it is common it could reduce take up of replay.

> For the purposes of this section, a "complete" transaction is one for
> which the server has made a final decision to accept or reject the
> message, whether or not it has managed to communicate this decision
> to the client. (This decision is the commit point.) A "partial"
> transaction is one for which the client has not received all the
> server's replies.

This sounds as though, while the accept/reject decision is in flight, the server considers the message "complete" but the client considers it "partial". I take it that that isn't what you mean.

[ Typo in 10.1 - s/REPLAT CLEAR/REPLAY CLEAR/

Really picky: Section 4, second bullet:
This means that at most one message can be duplicated if the connection is lost.
s/can/will/
]

[fanf]

Will the client be allowed to switch between smtps and starttls, or between ports (25, 587, 465), as long as they always use TLS ? I'm thinking about the case where my mobile ISP happily allows port 25 but the connection dies so I retry over my home ISP which blocks port 25 ?

From the IETF's point of view, smtps doesn't exist :-) But switching between ports 25 and 587 on the same submission server should be OK. I expect that actual implementations will treat 465 the same way as 587+STARTTLS.

Are message-size values typically accurate to the octet ? I remember bugs in exim where it wasn't always correctly updated - I think for header-rewrites. Clearly this is a problem with the submitting application, but if it is common it could reduce take up of replay.

A good point. Another area where byte counts get tricky is when using the Lemonade BURL extension, which allows you to compose a message on an IMAP server e.g. so that you can forward an attachment without downloading and uploading it again. The message is submitted by passing a self-authorizing URL to the message submission server which retrieves it from the IMAP server. (This avoids embedding SMTP+extensions within IMAP.) BURL can be combined with BDAT to provide the message header, new message content, and MIME framing for a forwarding attachment. I think I'll see what others have to say before tackling the problem :-)

This sounds as though, while the accept/reject decision is in flight, the server considers the message "complete" but the client considers it "partial". I take it that that isn't what you mean.

That's exactly what I mean. The discrepancy reflects the fact that different parts of a distributed can have different views of the status of a transaction and will need to roll back or forwards to become consistent. I'll add a clarification.

Thanks for the feedback.