Musings from a small IPP

Things have changed a lot in the 30 years I’ve been doing e-mail systems.  In those first heady days, establishing a transport fabric was hard work, and getting a message through over a mixture of uucp, decnet, snads, and arpa protocols a miracle of managing complexity, and co-operaion.  Now in February 2010 I reject 92% of the messages offered for my customers.  Strangely few of these rejections result in contacts from customers or their contacts  about false rejections — far more contacts relate to the SPAM we’re still letting through.   (continue reading…)

Sometimes I wonder when I get requests like this, especially from customers on a very basic hosting package.  Initially it might seem a perfectly reasonable request, but it does not work for multiple customers, and each customer’s follow-up is inevitably a request that would take several hours to achieve.

Now let’s just think what the outage might be, be it a router failure losing connexion, or an Internet storm, typically these things are about 3 hours duration.  If we are to arrange to call the customers, even at 5 minutes each, we’ll have barely started with just 36 calls per line, and have achieved nothing, by the time it all comes back.  If the failure is our equipment that time would be better spent configuring a replacement and getting it on site.

So the answer is “No, it’s not in your, or our other customers’ best interest.  If it really is an issue for you, would you like to talk about multi-site redundant servers?”

The only way the customer is going to get that sort of personal response is when their hosting fees go up by three orders of magnitude, effectively having their own dedicated full time support person.

“Ok so how do I make a big SPF record? I’ve run over the size of a DNS TXT record.”

Well as I commented elsewhere you can’t simply have a bunch of records as only the first one received will be applied, and if the software notices there’s more than one it’ll throw a wobbly. You can however split your spf record up like this:

$ORIGIN X.com.
@           IN      TXT     "v=spf1 a mx include:part1.x.com include:part2.x.com include:part3.x.com -all"
part1       IN      TXT     "v=spf1 ip4:192.168.128.0/20 -all"
part2       IN      TXT     "v=spf1 ip4:192.168.64.0/22 -all"
part3       IN      TXT     "v=spf1 ip4:192.168.192.0/18 -all"

Of course if you’re setting up for a large multinational, then it’s sensible to make the included parts correspond to your national gateways so you can use the a and mx tags, and avoid having to set up separate SPF records for national presentation.

I’ve been rehearsing that comment for a while, as people tell me they can’t send mail to my customers, and what am I going to do about it?

SPF is the Sender Policy Framework.  If a mail fails SPF that means that the sender (you)  has violated the policy set forth by the owner of the sending domain, or their agents, in an SPF record; the recipient is not an actor in this

By publishing an SPF record the sending domain’s owner has asked the world to make some simple checks that mail is coming from the correct place.  These checks help with avoiding being impersonated.

When a correct rejection happens, we never hear about it.  When a rejection hits someone who might legitimately expect to use the domain there are two possible causes, and one solution.

1. The mail is coming from the wrong place, such as sending work mail through a home ISP because the VPN to work has dropped, or perhaps there’s a new gateway at work which hasn’t been configured into the SPF.
   Another common case is trying to use an SPF protected address from public webmail providers who have not configured their servers in an SPF aware way.
2. Your domain’s SPF record is scrambled, so the receiving software can’t make a list of the places the mail is allowed to come from

The solution in both cases is to fix the SPF record, not to ask the recipient to special-case your mail.

There is a third case of websites doing the wrong thing which I’ve dealt with elsewhere

It’s amazing how easy it is to get something wrong when configuring complex systems. When it comes to networking the first thing that people notice is when their e-mail goes wrong, because e-mail is a fairly complex application these days that calls upon all of the underlying services in many unexpected ways. (continue reading…)

Some companies live on top of their hardware. When there is a failure it;s just a walk down the corridor. For OA5 it takes a bit longer, which is why we rent space from a hosting centre which does have such well placed technical guys, but manage most things ourselves with remote screen switches, and remote power switches.

Even with all that a particular piece of hardware may well still decide to go permanently out to lunch, and there is no alternative but to go to the backups to build a replacement system. This is where a near-current image of the user area really helps. Given the near-current image the rsync protocol is very efficient at bringing it forward to current as of last backup.

This does no mean we have spare hardware images of our major servers only XEN virtual machines ticking over which can be brought on line with the addition of a few IP addresses, to work on until a new piece of hardware can be prepared

Shortcuts make life easier for us.  For administrators configuring DNS there is a great shortcut which tells the program reading the zone where it is.  This lets the administrator leave off the domain part of the thing they’re configuring.

OK that  sounds complicated, so let’s give an example – if in a DNS file I were to write

$ORIGIN X.com.

then a few lines later I can say “mail” and “mail.X.com.” will be understood.  Well and good, though often a source of problems when someone leaves off a terminating ‘.’ and gets the domain added on where they were not expecting it.

SPF also has a chance to get messed up here.   Today’s gem was a record with

"v=spf1 mx a:ironport a:sandberg"

which makes one think the administrator setting it up was expecting that shorthand to apply to those ‘a:’ elements. It’s not clear what they thought they were doing for non-matching source addresses, as they left off a closure element.

"v=spf1 mx a:ironport.X.com a:sandberg.X.com -all"

was of course what they meant, but the software isn’t meant to follow inferences, rather it fails their SPF validation with a permanent error.

Today a customer wrote to ask for help getting mail from a company — let’s call it X — hat was suddenly not getting through.

X has 4 SPF rrcords

• v=spf1 mx include:isp1 -all
• v=spf1 mx include:isp2 -all
• v=spf1 mx include:isp3 -all
• v=spf1 mx include:isp4 -all

So depending on which of these four records is randomly received first, one of their supporting ISPs is authorized to forward X’s outbound email, — and — the other three ISPs are not authorized.   Actually X also had a syntax error in one of their 4 records which seriously did not help.

The SPF record X should have is of course:

• v=spf1 mx include:isp1 include:isp2 include:isp3 include:isp4 -all

Today I had a designer expressing the view that  mail rejection was the recipient’s “fault” not their web form.

They were using the user supplied e-mail address as the sender of the message generated by their web form.

The difference between Sender and From is that From is part of the headers, and is used as a reply address in the absence of a Reply-to header, whereas Sender is used in the SMTP transaction as the Mail From part.

Mail From/Sender is often different from From — eg when a secretary sends on behalf of their boss, and this is reflected in SMTP email with that distinction.

It is an issue these days as there is an anti-SPAM mechanism called SPF (Sender Policy Framework) [q.v. for details], but in summary it allows the owners of a domain to say ‘Legitimate mail from our domain only comes from these places” (usually their mail server, and web server), and add, “if it comes from elsewhere discard it as SPAM’ — now the thing to notice here is it is a tag that applies to the sender address (Mail From in SMTP), and may have nothing whatsoever to do with the recipient who is being protected from SPAM, and the owners of a domain are not going to authorise any and all random websites to use their domain to send mail as their company, just so that it can reply to them.

Gmail and hotmail for instance use the SPF mechanism to protect their customers of being accused of spamming, so the constituency your form will fail for is fairly large

This is why any web form you create has to use a legitimate sending address for that server, even if you put the address of the form-filler into the From field where it will be displayed to the recipient. It also means you are taking responsibility for any abuse of forms on your server.

I love XEN virtual machines, they take up so much less rack space, but I’m probably missing a trick or two in terms of backing them up.

Today’s big job is recovering from whatever hit the hosting centre at 15:15 yesterday.  It knocked sideways a couple of big disks.  One was on the main server, a SAS disk that appears to have recovered for now, but has been promoted up the preventative maintenance stack, and needs to be replaced in due course.  This was operating with Software Raid mirroring.

The other big disk was a 1T (920G really — I wish suppliers would work in base 2 like the rest of th computer industry, and that the marketing types who set out to intentionally mislead with this sort of flummery could be censured, but it will never happen, as the advertising standards authority seems to be staffed with innumerate arts graduates).  Anyway this disk is Serial ATA, and was supposed to be running under the on-motherboard hardware RAID.  Well as one of the meanings for supposed is ‘fondly believed’ that’s about right.

The disk containing the mirror was largely bare.  Of course the main disk is complete Toast, so we’re bringing stuff back from backup onto a spare machine.   To make matters more interesting the toasted machine had 7 virtual servers on board, which makes the restore exercise that much larger.  It’s been a long night

And then we have to understand what went wrong, and take steps to prevent repetition.  We await feedback from the hosting centre.