Save the US Postal Service offices!

This is a corker. Today, while wandering past a Staples store in the South Bay I saw a bunch of people outside waving protest signs. Upon closer examination this turned out to be Stop Staples!, a campaign by the American Postal Workers Union (motto "Don't mention Seinfeld"). Staples is the same kind of store in the USA as it is in the UK, providing all kinds of stationery and office supplies. Since late 2013 the US Postal Service has been running a trial program with post office counters in Staples stores, staffed by Staples workers rather than APWU-unionized US Postal Service employees. Hence the protest. The APWU seems rather concerned that the trial program is about to expand.

Reading the AWPU background briefing on the protest is illuminating and amusing in roughly equal measures for anyone who has ever spent time in a US post office:

"Staples and USPS management are perpetrating a fraud on the people of this country," says APWU President Mark Dimondstein. "They are promoting the deal as though taking your mail to Staples is the same as taking it to the Post Office. "It’s not."
He's right, you know. If I go to Staples to make a purchase at the postal counter then I can reasonably expect to be in and out in ten minutes. For the regular post office - once I can find it and get a parking space - I'm budgeting a full hour and bringing a book.
Staples' low-paid, high-turnover employees get just four hours of "classroom" training for postal retail duties.
I don't know about "low-paid". In California they're subject to the state minimum wage which is $9/hour now and $10/hour from January 1st, and there seems to be a thriving demand for competent retail employees. And if the US Postal Service is paying as much as McDonald's for most of their counter staff, they - or rather, the US taxpayer who's funding them - are getting a really bad deal.
Postal workers must pass a test before they are considered qualified to work the window
I can only imagine that it involves the examiner locating a pulse on the worker, with a generous margin for error.

What this is about, of course, is that the APWU is terrified of its membership shrinking, and the associated contributions to the existing retirement plans falling. The USPS retirement plan (healthcare and pensions) funding is in a horrendously bad state as it is, and shrinking the operations, staffing and funding of the USPS will make this situation even clearer, the gap harder to plug, and the public less inclined to back additional federal spending to fill the hole. "Why do I care about the local post office? I go to Staples when I want to post something." The USPS is going to be left with just local letter delivery after Fedex and UPS takes the profitable parcel delivery, and the bulk of those letters are junk mail that the USPS loves for the money and the recipients hate for the spam.

Amusingly, around midday the protestors all left en masse. Presumably they were on their lunch break, a staple feature of US post offices in my experience. As soon as the lunchtime queues start to build up, the counter staff react by closing several of the open counters and wandering off, presumably to have a leisurely lunch. If they've got any eye to the future, I hope they're dusting off their resumés and looking to move to a counter position at Staples before the rush.

Let me quote the APWU leaflet again, in closing:

During the first quarter of Fiscal Year 2014 the USPS enjoyed an operating surplus of $765 million. But the agency’s good news was buried in most media accounts, which said the USPS suffered a loss of $354 million loss. The USPS reported losses for the first quarter of 2014 for one reason – the congressional mandate that requires the Postal Service to pre-fund healthcare benefits for future retirees.
Indeed, damn Congress for insisting that government businesses track their accumulated liabilities as well as their income...


The BBC asks "do people become more prejudiced as they age?"


Psychologists used to believe that greater prejudice among older adults was due to the fact that older people grew up in less egalitarian times. In contrast to this view, we have gathered evidence that normal changes to the brain in late adulthood can lead to greater prejudice among older adults.
There are certainly normal changes to the brain. We call that "life".

Old people have experienced more of life than when they're young, so they have more facts at their disposal to make judgements.

This isn't prejudice. It's postjudice. So the BBC approach of venerating the young and disapproving of the attitudes of older generations is precisely the wrong way around.


The spectacular kind of hardware failure

Gentle reader, I have attempted several times to pen my thoughts on the epic hack of the US Office of Personnel Management that compromised the security information of pretty much everyone who works for the US government, but I keep losing my vision and hearing a ringing in my ears when I try to do so. So I turn to a lesser-known and differently-awesome fail: the US visa system.

Since a computer failure on the 26th of May - over three weeks ago - the US embassies and consulates worldwide have been basically unable to issue new visas except in very limited circumstances. You haven't heard much about this because it hasn't really affected most US citizens, but believe me it's still a big issue. It seems that they're not expecting the system to be working again until next week at the earliest. Estimates of impacted users are on the order of 200,000-500,000; many people are stuck overseas, unable to return to the USA until their visa renewal is processed.

What happened? The US Department of State has a FAQ but it is fairly bland, just referring to "technical problems with our visa systems" and noting "this is a hardware failure, and we are working to restore system functions".

So a hardware failure took out nearly the entire system for a month. The most common cause of this kind of failure is a large storage system - either a mechanical failure that prevents access to all the data you wrote on the disks, or a software error that deleted or overwrote most of the data on there. This, of course, is why we have backups - once you discover the problem, you replace the drive (if broken) and then restore your backed up data from the last known good state. You might then have to apply patches on top to cover data that was written after the backup, but the first step should get you 90%+ of the way there. Of course, this assumes that you have backups and that you are regularly doing test restores to confirm that what you're backing up is still usable.

The alternative failure is of a relatively large machine. If you're running something comparable to the largest databases in the world you're going to be using relatively custom hardware. If it goes "foom", e.g. because its motherboard melts, you're completely stuck until an engineer can come over with the replacement part and fix it. If the part is not replaceable, you're going to have to buy an entirely new machine - and move the old one out, and install the new one, and test it, and hook it up to the existing storage, and run qualification checks... But this should still be on the order of 1 week.

A clue comes from a report of the State Department:

"More than 100 engineers from the government and the private sector [my emphasis] are working around the clock on the problem, said John Kirby, State Department spokesman, at a briefing on Wednesday.
You can't use 100 engineers to replace a piece of hardware. They simply won't fit in your server room. This smells for all the world like a mechanical or software failure affecting a storage system where the data has actually been lost. My money is on backups that weren't actually backing up data, or backing it up in a form that needed substantial manual intervention to restore, e.g. a corrupted database index file which would need every single piece of data to be reindexed. Since they've roped in private sector engineers, they're likely from whoever supplied the hardware in question: Oracle or IBM, at a guess.

The US Visa Office issues around 10 million non-immigrant visas per year, which are fairly simple, and about 500,000 immigrant visas per year which are a lot more involved with photos, other biometrics, large forms and legal papers. Say one of the latter takes up 100MB (a hi-res photo is about 5MB) and one of the former takes up 5MB; then that's a total of about 100TB per year. That's a lot of data to process, particularly if you have to build a verification system from scratch.

I'd love to see a report on this from the Government Accountability Office when the dust settles, but fear that the private sector company concerned will put pressure on to keep the report locked up tight "for reasons of commercial confidentiality and government security". My arse.


Courageous journalism at the BBC

I kid, obviously. When describing the current controversy over the Washington D.C. Metro refusing to take any "issue-oriented" adverts until next year just so that they can avoid showing the prize-winning "Draw Mohammed" cartoon, the BBC resorts to words rather than a picture to describe the salient image.

The advert calls for Americans to support free speech and features a bearded, turban-wearing Muhammad waving a sword and shouting: "You can't draw me!"
In reply, a cartoon bubble portrays an artist grasping a pencil and saying: "That's why I draw you."
How odd, you would have thought that they would have included an image of the cartoon rather than laboriously describe its contents.

Just to make the point, here's the image in question:

The spineless BBC writer isn't shy of displaying their orientation towards issues:

Ms Geller insists the cartoon is a "political opinion" which does not contain any violence.
Ms Geller is of course correct. There's no violence in that picture: the gentleman depicted is holding a sword, but that's as far as it goes. Yet the writer takes particular care to use reported speech and quotes, presumably to demonstrate that he or she is emphatically not in sympathy with Ms Geller or (mysteriously unnamed in the article) artist Bosch Fawstin.

Deary me. Truely, the BBC has resigned from actual journalism in order to be at the back of the line when crocodile feeding time comes around.

I'm really not keen on Pamela Gellar, but the rest of the world seems to be bending over backwards to make her admittedly extreme opinions seem really quite rational and reasonable. And we are surprised when Muslim extremism is emboldened by this obvious cowardice?


Delays are good for you - the MTA proves it

No, really, they do. New York's Metropolitan Transit Authority (something like Transport for London) has produced an outstanding video that shows why making some subway trains late makes others less late:

Yes, the idea is that sometimes delaying a train can prevent further delays by not compounding the gap between trains. Anyone who has waited impatiently on a hot subway platform might find this concept counterintuitive, but transportation experts generally agree that that the evenness of service is as crucial as avoiding individual delays.
The MTA video makes a compelling case. The key insight is that once a platform gets crowded enough, due to constant feed of new passengers and a delayed train, it becomes slower for the next train to debark and embark passengers. So an already delayed train gets more delayed as it progresses down the line. The solution? Spot a train that's getting near the critical delay time and give it priority to progress through the network even if this involves delaying other (less delayed trains).

It's a great example that, even in what we regard as relatively simple systems, there can be a complex interplay between entities that produce highly unintuitive results. Deliberately delaying trains can actually be good for the system as a whole (if not for the passengers sitting in the delayed train with their faces pressed into a fellow passenger's unwashed armpit).


You should care about moving to HTTPS

Eric Mill's "We're Deprecating HTTP and it's going to be okay" is a must-read call-to-arms for everyone with a site on the Internet, explaining why the transition from unencrypted web traffic (HTTP) to encrypted (HTTPS) is actually fundamental to the future existence of the democratic web-as-we-know it.

For the 90% of my reading audience who are already saying "Bored now!" here's why it matters to you. Sir Tim Berners-Lee invented HTTP (the language of communication between web browser and web server) in CERN, a European haven of free thought, trust and international co-operation. The 1930s idea that "Gentlemen do not read each other's mail" was - surprisingly, given the history of cryptographic war in WW2 - fundamental to HTTP; messages might have transited systems owned by several different groups, but none of them would have thought to copy the messages passing through their system, let alone amend them.

This worked fine as long as no-one was interested in the communication of harmless nerds about their hobbies, much as the government-owned Royal Mail doesn't bother to copy the contents of postcards passing through their sorting offices because they only contain inane drivel about sun, sea and sand. However, once people realized that they could communicate freely about their occasionally subversive ideas across borders and continents, and financial institutions woke to the possibility of providing services without paying for expensive un-scalable fallible human cashiers, many governments and other less-legal entities wanted to read (and sometimes alter) Internet traffic.

Mills gives two great examples of where HTTPS prevented - and could have prevented further - nation-state abuse of Internet content:

- The nation of India tried and failed to ban all of GitHub. HTTPS meant they couldn't censor individual pages, and GitHub is too important to India's tech sector for them to ban the whole thing.
- The nation of China weaponized the browsers of users all over the world to attack GitHub for hosting anti-censorship materials (since like India, they can't block only individual pages) by rewriting Baidu's unencrypted JavaScript files in flight.
And closer to home, Cameron's plan to make all online communication subject to monitoring is so stupidly illiberal and expensively pointless that it deserves to be made impractical by general adoption of HTTPS. GCHQ and friends can tap all the Internet traffic they like: if it's protected by HTTPS, the traffic is just taking up disk space to no practical purpose. Brute-forcing, even with nation-state resources, is so expensive that it's reserved for really high-value targets. GCHQ would have to go after something fundamental like a Certificate Authority, which would leave big and obvious fingerprints, or compromise a particular user's machine directly, which doesn't scale.

As long as users are still relaxed about the absence of a padlock in their browser bar, HTTP will continue to provide a route for governments to snoop on their citizens' traffic. So let's give up on HTTP - it has had its day - and move to a world where strongly encrypted traffic is the default.


You can't be too careful - car crashes

The class of systems with high distributed costs and focused but inadequate benefits is going to have another member: auto-calling police in the event of a car crash:

In the event of a crash, the device calls the E.U.'s 911 equivalent (112) and transmits to authorities important information including location, time, and number of passengers in the vehicle. An in-car button will also be installed in all vehicles. The eCall requirement will add an estimated $100 to the price of a car.
$100 on each (new) car sold: so how many new cars are sold in the EU each year? About 14 million in 2012. So this measure will cost $1.4 billion, and maybe $150 million in the UK. What's the benefit?
Each year nearly 26,000 people are killed in the E.U. by car crashes. This new device is estimated to reduce that number by 10 percent, saving 2,600 lives annually, by cutting down emergency response time by as much as 60 percent.
The cost of a life for purposes of safety varies by country and mode of transport, but let's take $1 million as the average. Given the quoted statistics, $2.6 billion saving (though optimistic, probably lower) comprehensively dwarfs $1.4 billion cost (though also optimistic, probably higher). Why isn't this a slam-dunk decision?

The problem is twofold: a) zeroing cost for lives saved, and b) the assumption of 10% saving. Let's consider each in turn.

If an injury is potentially fatal but not actually fatal due to timely intervention, it's almost always due to either early suppression of severe blood loss, or timely (within 1-2 mins) clearing of obstructed airway. The latter isn't relevant due to emergency service response times, so we only consider the former. This injured person will still need emergency treatment followed by several days of hospital care, and quite possibly follow-on care of injuries, rehab, and in some cases reduced lifetime tax payments due to reduced earnings and disability payments, so you're looking at order of $100K average costs. That's still not really significant.

However, consider a typical case where a life is saved: a car driver has an accident in the countryside when no-one is around. His car calls 112 and so the police (not the ambulance service initially, because they are too stretched to respond to wild goose chases) respond to his location. Seeing the crash they call for an ambulance which arrives 10-30 minutes before it would have otherwise arrived due to a passer-by report - people tend to notice a crashed car with no emergency services around it. He would have died due to shock (depletion of oxygen to the critical organs due to blood loss / asphyxiation / traumatic damage to heart and lungs) but the ambulance got there in time to oxygenate him and transport to hospital. Just how common is this?

Fatal road accidents rarely happen on remote roads - unsurprisingly, they happen where there are many more cars and roadside obstructions to run into. If an accident happens where passers-by are prevalent, this system doesn't help at all since nearly all passers-by have mobile phones. So we're only looking at a small fraction - 5% is optimistic - of accidents. The press release assumed 10%, so the benefit has already halved and is perilously close to the cost.

But bleeding to death is not a common cause of death from road accidents for drivers/passengers. Much more likely is traumatic head injury, which tends to kill them right there in the car. Unsecured drivers/passengers fly through the windscreen, or secured drivers/passengers bang their head against the car frame. This kills instantly, or in a few minutes. Another mechanism is the "third collision" where the car bangs into a tree (collision 1), the driver bangs into their seatbelt (collision 2) and then the free-hanging organs like lungs, heart bang into the drivers chest, or their blood vessels bang into ligaments that cheesewire them (collision 3). If you're in this situation and your aorta (the major blood vessel coming out of the heart) is damaged you can expect a 60%-80% chance of death no matter how quickly you get to the hospital.

Therefore, before we stick the European population with an extra $1 billion of annual costs, why don't we conduct a limited experiment introducing this requirement into a single country which is similar to another country in road crash death rates to see what effect, if measurable, this measure has? Or is the notion of trade-offs too alien to the EU?


Journos writing about trading and high-speed computing

I have to admit, this amused me - the Daily Mail trying to write about high-frequency trading:

Suspected rogue trader Navinder Sarao lived in his parents' modest home because it gave him a split-second advantage worth millions of pounds, it was claimed yesterday.
His family's semi-detached house in suburban West London is closer to an internet server used by one of the major financial exchanges, giving him a nanosecond advantage over rivals in the City.
Sarao, 36, was dubbed the 'Hound of Hounslow' after it emerged he lived at home with his parents, despite allegedly making £26.7million in just four years of dealing from their home.
And yet you'd think that renting a small flat in Slough and paying for Internet access there would have improved his speed advantage; at a cost of about £50K for four years, that would have been a bargain. Why, it's almost as if the Daily Mail journalists had no idea what they were talking about....


Active attack on an American website by China Unicom

I wondered what the next step in the ongoing war between Western content and Chinese censorship might be. Now we have our answer.

"Git" is a source code repository system which allows programmers around the world to collaborate on writing code: you can get a copy of a software project's source code onto your machine, play around with it to make changes, then send those changes back to Git for others to pick up. Github is a public website (for want of a more pedantic term) which provides a repository for all sorts of software and similar projects. The projects don't actually have to be source code: anything which looks like plain text would be fine. You could use Github to collaborate on writing a book, for instance, as long as you used mostly text for the chapters and not e.g. Microsoft Word's binary format that makes it hard for changes to be applied in sequence.

Two projects on Git are "greatfire" and "cn-nytimes" which are, respectively, a mirror for the Greatfire.org website focused on the Great Firewall of China, and a Chinese translation of the New York Times stories. These are, obviously, not something to which the Chinese government wants its citizenry to have unfettered access. However, Github has many other non-controversial software projects on it, and is actually very useful to many software developers in China. What to do?

Last week a massive Distributed Denial of Service (DDoS) attack hit Github:

The attack began around 2AM UTC on Thursday, March 26, and involves a wide combination of attack vectors. These include every vector we've seen in previous attacks as well as some sophisticated new techniques that use the web browsers of unsuspecting, uninvolved people to flood github.com with high levels of traffic. Based on reports we've received, we believe the intent of this attack is to convince us to remove a specific class of content. [my italics]
Blocking Github at the Great Firewall - which is very easy to do - was presumably regarded as undesirable because of its impact on Chinese software businesses. So an attractive alternative was to present the Github team with a clear message that until they discontinued hosting these projects they would continue to be overwhelmed with traffic.

If this attack were just a regular DDoS by compromised PCs around the world it would be relatively trivial to stop: just block the Internet addresses (IPs) of the compromised PCs until traffic returns to normal levels. But this attack is much more clever. It intercepts legitimate requests from worldwide web browsers for a particular file hosted on China's Baidu search engine, and modifies the request to include code that commands repeated requests for pages from the two controversial projects on Github. There's a good analysis from NetreseC:

In short, this is how this Man-on-the-Side attack is carried out:
1. An innocent user is browsing the internet from outside China.
2. One website the user visits loads a JavaScript from a server in China, for example the Badiu Analytics script that often is used by web admins to track visitor statistics (much like Google Analytics).
3. The web browser's request for the Baidu JavaScript is detected by the Chinese passive infrastructure as it enters China.
4. A fake response is sent out from within China instead of the actual Baidu Analytics script. This fake response is a malicious JavaScript that tells the user's browser to continuously reload two specific pages on GitHub.com.

The interesting question is: where is this fake response happening? We're fairly sure that it's not at Baidu themselves, for reasons you can read in the above links. Now Errata Security has done a nice bit of analysis that points the finger at the Great Firewall implementation in ISP China Unicom:

By looking at the IP addresses in the traceroute, we can conclusive prove that the man-in-the-middle device is located on the backbone of China Unicom, a major service provider in China.
That existing Great Firewall implementors have added this new attack functionality fits with Occam's Razor. It's technically possible for China Unicom infrastructure to have been compromised by patriotically-minded independent hackers in China, but given the alternative that China Unicom have been leant on by the Chinese government to make this change, I know what I'd bet my money on.

This is also a major shift in Great Firewall operations: this is the first major case I'm aware of that has them focused on inbound traffic from non-Chinese citizens.

Github look like they've effectively blocked the attack, after a mad few days of scrambling, and kudos to them. Now we have to decide what the appropriate response is. It seems that any non-encrypted query to a China-hosted website would be potential fair game for this kind of attack. Even encrypted (https) requests could be compromised, but that would be a huge red arrow showing that the company owning the original destination (Baidu in this case) had been compromised by the attacker: this would make it 90%+ probable that the attacker had State-level influence.

If this kind of attack persists, any USA- or Europe-focused marketing effort by Chinese-hosted companies is going to be thoroughly torpedoed by the reasonable expectation that web traffic is going to be hijacked for government purposes. I wonder whether the Chinese government has just cut off its economic nose to spite its political face.


What does "running your own email server" mean?

There's lots of breathless hyperbolae today about Hillary Clinton's use of a non-government email address during her tenure as Secretary of State. The Associated Press article is reasonably representative of the focus of the current debate:

The email practices of Hillary Rodham Clinton, who used a private account exclusively for official business when she was secretary of state, grew more intriguing with the disclosure Wednesday that the computer server she used traced back to her family's New York home, according to Internet records reviewed by The Associated Press.
It was not immediately clear exactly where Clinton's computer server was run, but a business record for the Internet connection it used was registered under the home address for her residence in Chappaqua, New York, as early as August 2010. The customer was listed as Eric Hoteham.
Let's apply a little Internet forensics to the domain in question: clintonemail.com. First, who owns the domain?
$ whois clintonemail.com
Registry Domain ID: 1537310173_DOMAIN_COM-VRSN
Registrar WHOIS Server: whois.networksolutions.com
Registrar URL: http://networksolutions.com
Updated Date: 2015-01-29T00:44:01Z
Creation Date: 2009-01-13T20:37:32Z
Registrar Registration Expiration Date: 2017-01-13T05:00:00Z
Registrar IANA ID: 2
Registrar Abuse Contact Email: abuse@web.com
Registrar Abuse Contact Phone: +1.8003337680
Domain Status:
Registry Registrant ID:
Registrant Organization:
Registrant Street: 12808 Gran Bay Parkway West
Registrant City: Jacksonville
Registrant State/Province: FL
Registrant Postal Code: 32258
Registrant Country: US
Registrant Phone: +1.5707088780
Registrant Phone Ext:
Registrant Fax:
Registrant Fax Ext:
Registrant Email: kr5a95v468n@networksolutionsprivateregistration.com
So back in January this year the record was updated, and we don't necessarily know what it contained before that, but currently Perfect Privacy, LLC are the owners of the domain. They register domains on behalf of people who don't want to be explicitly tied to that domain. That's actually reasonably standard practice: any big company launching a major marketing initiative wants to register domains for their marketing content, but doesn't want the launch to leak. If Intel are launching a new microbe-powered chip, they might want to register microbeinside.com without their competitors noticing that Intel are tied to that domain. That's where the third party registration companies come in.

The domain record itself was created on the 13th of January 2009, which is a pretty strong indicator of when it started to be used. What's interesting, though, is who operates the mail server which receives email to this address. To determine this, you look up the "MX" (mail exchange) records for the domain in question, which is what any email server wanting to send email to hillary@clintonemail.com would do:

$ dig +short clintonemail.com MX
10 clintonemail.com.inbound10.mxlogic.net.
10 clintonemail.com.inbound10.mxlogicmx.net.
mxlogic.net were an Internet hosting company, bought by McAfee in 2009. So they are the ones running the actual email servers that receive email for clintonemail.com and which Hillary's email client (e.g. MS Outlook) connected to in order to retrieve her new mail.

We do need to take into account though that all we can see now is what the Internet records point to today. Is there any way to know where clintonemail.com's MX records pointed to last year, before the current controversy? Basically, no. Unless someone has a hdr22@clintonemail.com mail from her home account which will have headers showing the route that emails took to reach her, or has detailed logs from their own email server which dispatched an email to hdr22@clintonemail.com, it's probably not going to be feasible to determine definitively where she was receiving her email. However, CBS News claims that the switch to mxlogic happened in July 2013 - that sounds fairly specific, so I'll take their word for it for now. I'm very curious to know how they determined that.

All of this obscures the main point, of course, which is that a US federal government representative using a non-.gov email address at all for anything related to government business is really, really bad. Possibly going-to-jail bad, though I understand that the specific regulation requiring a government employee to use a .gov address occurred after Hillary left the role of SecState (Feb 2013). Still, if I were the Russian or Chinese foreign intelligence service, I'd definitely fancy my chances in a complete compromise of either a home-run server, or of a relatively small-scale commercial email service (mxlogic, for instance).

Desperately attempting to spin this whole situation is Heidi Przybyla from Bloomberg:

OK, let's apply our forensics to jeb.org:
$ dig +short jeb.org MX
5 mx1.emailsrvr.com.
10 mx2.emailsrvr.com.
emailsrvr.com is, like mxlogic.net, a 3rd party email hosting service, apparently specialising in blocking spam. I'm not surprised that someone like Jeb Bush uses it. And, like Hillary, he isn't "running his own email server", he's using an existing commercial email server. It's not Gmail/Outlook.com/Yahoo, but there's not reason to think it's not perfectly serviceable, and it's not controlled by Bush so if they log or archive incoming or outgoing email his correspondence is legally discoverable.

The difference between Jeb Bush and Hillary Clinton of course, as many others note, is that Jeb is not part of the US federal government and hence not subject to federal rules on government email...