Musings from a small IPP

It’s a very sad milestone.  As of this week we mark over 98% of the messages we are offered as SPAM.  The press has some catching up to do — they report 92% — a figure we were at back in February.

This does include a very small percentage of messages which we later re-classify as ‘ham,’ and the slightly larger set that we have mistakenly believed to be legitimate, but the percentage is scary.  Handling it would require 1 processor per 750 mailboxes running flat out (on average — the peaks and troughs are of course rather fractal), meaning that a large multi-processor system is required in practice.

So less than 1 message in 50 is legitimate.

When I look at this figure, I realise we do something really worthwhile for our customers.

Things have changed a lot in the 30 years I’ve been doing e-mail systems.  In those first heady days, establishing a transport fabric was hard work, and getting a message through over a mixture of uucp, decnet, snads, and arpa protocols a miracle of managing complexity, and co-operaion.  Now in February 2010 I reject 92% of the messages offered for my customers.  Strangely few of these rejections result in contacts from customers or their contacts  about false rejections — far more contacts relate to the SPAM we’re still letting through.   (continue reading…)

Sometimes I wonder when I get requests like this, especially from customers on a very basic hosting package.  Initially it might seem a perfectly reasonable request, but it does not work for multiple customers, and each customer’s follow-up is inevitably a request that would take several hours to achieve.

Now let’s just think what the outage might be, be it a router failure losing connexion, or an Internet storm, typically these things are about 3 hours duration.  If we are to arrange to call the customers, even at 5 minutes each, we’ll have barely started with just 36 calls per line, and have achieved nothing, by the time it all comes back.  If the failure is our equipment that time would be better spent configuring a replacement and getting it on site.

So the answer is “No, it’s not in your, or our other customers’ best interest.  If it really is an issue for you, would you like to talk about multi-site redundant servers?”

The only way the customer is going to get that sort of personal response is when their hosting fees go up by three orders of magnitude, effectively having their own dedicated full time support person.

Some companies live on top of their hardware. When there is a failure it;s just a walk down the corridor. For OA5 it takes a bit longer, which is why we rent space from a hosting centre which does have such well placed technical guys, but manage most things ourselves with remote screen switches, and remote power switches.

Even with all that a particular piece of hardware may well still decide to go permanently out to lunch, and there is no alternative but to go to the backups to build a replacement system. This is where a near-current image of the user area really helps. Given the near-current image the rsync protocol is very efficient at bringing it forward to current as of last backup.

This does no mean we have spare hardware images of our major servers only XEN virtual machines ticking over which can be brought on line with the addition of a few IP addresses, to work on until a new piece of hardware can be prepared

There are 2 really lovely pieces of contributer-supported software out there from an IPPs perspective.  Both have a really lovely property — they can be installed as ‘multisite’ configurations, that is one only needs to install the software once to have many users able to use it to create their own sites.  Within certain limits. (continue reading…)

Shortcuts make life easier for us.  For administrators configuring DNS there is a great shortcut which tells the program reading the zone where it is.  This lets the administrator leave off the domain part of the thing they’re configuring.

OK that  sounds complicated, so let’s give an example – if in a DNS file I were to write

$ORIGIN X.com.

then a few lines later I can say “mail” and “mail.X.com.” will be understood.  Well and good, though often a source of problems when someone leaves off a terminating ‘.’ and gets the domain added on where they were not expecting it.

SPF also has a chance to get messed up here.   Today’s gem was a record with

"v=spf1 mx a:ironport a:sandberg"

which makes one think the administrator setting it up was expecting that shorthand to apply to those ‘a:’ elements. It’s not clear what they thought they were doing for non-matching source addresses, as they left off a closure element.

"v=spf1 mx a:ironport.X.com a:sandberg.X.com -all"

was of course what they meant, but the software isn’t meant to follow inferences, rather it fails their SPF validation with a permanent error.

I love XEN virtual machines, they take up so much less rack space, but I’m probably missing a trick or two in terms of backing them up.

Today’s big job is recovering from whatever hit the hosting centre at 15:15 yesterday.  It knocked sideways a couple of big disks.  One was on the main server, a SAS disk that appears to have recovered for now, but has been promoted up the preventative maintenance stack, and needs to be replaced in due course.  This was operating with Software Raid mirroring.

The other big disk was a 1T (920G really — I wish suppliers would work in base 2 like the rest of th computer industry, and that the marketing types who set out to intentionally mislead with this sort of flummery could be censured, but it will never happen, as the advertising standards authority seems to be staffed with innumerate arts graduates).  Anyway this disk is Serial ATA, and was supposed to be running under the on-motherboard hardware RAID.  Well as one of the meanings for supposed is ‘fondly believed’ that’s about right.

The disk containing the mirror was largely bare.  Of course the main disk is complete Toast, so we’re bringing stuff back from backup onto a spare machine.   To make matters more interesting the toasted machine had 7 virtual servers on board, which makes the restore exercise that much larger.  It’s been a long night

And then we have to understand what went wrong, and take steps to prevent repetition.  We await feedback from the hosting centre.

Last Wednesday Sweden went live with it’s anti file sharing law. This enables the content owners to demand tracebacks to those serving, and those downloading copyright material.

The traffic at the main swedish internet exchange dropped to 2/3 Tuesday’s value, so a significant cost saving you might think. It isn’t really though as Internet traffic volume is essentially fractal over time with huge peaks and troughs.

The interesting thing will be to see how CD/DVD sales and legitimate download volumes change. I’ll also be interested to see how often legitimate users of bit-torrent are falsely accused when they fetch a fresh CentOS distribution