The Silicon Valley Diversity Shitstorm

Talk about putting the cat among the pigeons. Journo Brian S. Hall wrote a short article about diversity in Silicon Valley: specifically "There Is No Diversity Crisis In Silicon Valley". It turns out to have been slightly controversial, as you can tell when you visit the original Forbes post:

The piece previously at this URL, titled "There Is No Diversity Crisis In Silicon Valley," published on 10/5/2015 [5th October 2015 for anyone using a sane date format], was deemed to have violated our Terms of Service and was removed.
Well, that's odd. What did it say? What could have violated the ToS?

Luckily, we can now read the original article on Brian's own site. An excerpt:

Silicon Valley doesn't just create greatness, it's probably the most open, welcoming, meritocratic-based region on the planet. Anecdotal evidence strongly suggests that disproportionately more Chinese, Indians, and LGBQT succeed in Silicon Valley than just about any place in America. Guess what? Everyone earned their job because of their big brains and ability to contribute.
The piece, to be fair, was rather heavy on anecdote. However in the ensuing Twitter blitzkreig there was some actual data posted including an illustrative stat on student study and degree achievements in STEM subjects which showed pretty clearly that white, Hispanic and black enroll in STEM programmes at about the same rate, but that the white students are disproportionately more successful in actually obtaining a degree. Asian students - of course! - enroll at twice the rate and obtain a degree disproportionately more often. So if you view a STEM degree as somewhat important in a Silicon Valley career then it's not entirely surprising that the ethnic makeup in SV correlates with those stats.

Anecdotally, Hall's assertions on race and LGBTQ seem about right to me, though I think he's missed a few letters off the latter term. The native Californians and other white Americans are distinctly in the numerical minority, and even obvious LGBTQ engineers are relatively plentiful. I also liked Hall's dig at the humanities as a contrast to "computer programming, engineering, chemistry — hard subjects that demand hard work", remembering the geography and history students lounging around after a couple of Finals exams early in the last semester after putting in a few strenuous 10-12 hour weeks, while the maths, physics and engineering students were still sweating away with 50 hour weeks revising for a series of painfully hard and objectively marked exams right at the end of the semester.

I'd probably take some issue with Hall's assertion that "Everyone earned their job because of their big brains and ability to contribute" - the latter is more aspirational than fact, SV hiring like anywhere else still has problems trying to determine whether someone who's obviously smart can actually be productive, and screws up that assessment reasonably frequently, but the basic idea is there. A SV company that does any discrimination other than by ability to do the job is going to shut off some of its source of talent, and in a hugely competitive hiring market that's a pretty dumb play.

Hall subsequently doubled down with the tweet

It's worth noting that Google CEO Pichai came from a poor family in India and rose based on brutal merit. Though he had problems with the humanities in school:
He was a brilliant student but his geography and history used to let him down so he was never top of the year.
That's probably what annoyed all the humanities graduates who piled onto Hall on Twitter... Although it's a little tricky to argue for a glass ceiling for minorities in SV given the above facts, it didn't seem to stop a lot of people from trying.

SV still has recruiting and retention problems, and I'd call out the experience of women in particular - the tendency of male engineers to act like baboons is off-putting to any women engineers who want to be something other than male engineers with a slightly different placement of genitals. But I don't find anything particularly jarring, scandalous or untrue in Hall's piece, so I wonder why exactly Forbes decided to withdraw it under pressure. If it wasn't pulled because of falsehoods, was it pulled because it was too true?


Some illegals more equal than others - California edition

In a conversation at work today, a colleague mentioned that her Iceland-born spouse needed someone to go with him to the local branch of the California Department of Motor Vehicles (aka the First Circle of Hell) because he had to take a test. There was widespread surprise at this - didn't he have a valid licence from another country, and wasn't this OK? Yes he did, and no it wasn't; as of 15th May, the California DMV will no longer issue temporary driving licences when you pass their written test.

For context on why this matters: for foreign citizens, when you move to California and become resident (paying rent / utility bills locally) you're required to get a driving licence within 10 days of this event if you want to continue driving in California. Up until May, this was straight forward: you went to the DMV, took their written tests - tedious but not too hard - then booked a practical test and in return got a temporary driving licence that you could renew if the test got postponed. The practical test took 1-3 weeks to reserve a reasonable slot until recently, but this year's announcement that certain immigrants didn't have to prove any legal residence status has caused a huge rush of applications and backlog of tests.

Now that foreign citizens don't get the temporary licence, they can't drive unaccompanied from day 11 of their residency until the date that they pass the (admittedly easy) driving test. Sounds like a bit of a regression, so what's going on?

Let's look at the requirements for California DMV form AB60 guidelines on proving identity if you're not already a Californian:

Foreign Document that is valid, approved by the Department of Motor Vehicles (DMV) and electronically verified by DMV with the country of origin:
  • Mexican Federal Electoral Card (Instituto Federal Electoral (IFE) Credencial para Votar – 2013 version)
  • Mexican Passport (issued in 2008 or later and includes digital photo and digital signature)
  • Mexican Consular Card (Matricula Consular – 2006 and 2014 versions)
  • Foreign Passport that is valid and approved by DMV (see page 4 & 5 for list of DMV approved passports). The customer must also provide his/her social security number (SSN) that is electronically verifiable with the Social Security Administration.

Well, that's tough luck if you're an illegal immigrant (i.e. not able to get a legit Social Security number because you're not a legal resident) and not Mexican, right? Luckily there's an alternative if you have a foreign passport but not an SSN: if you have one of the following then you're OK:
  • Argentinian Identification Card (Documento Nacional de Identidad (DNI) – 2009 or 2012 version)
  • Brazilian Consular Card (Carteira de Matricula Consular – 2010 version)
  • Chilean Identification Card (Cedula de Identidad – 2013 version)
  • Colombian Consular Card (Consular Registration – 2015 version)
  • Ecuadorian National Identification Card (Cedula de Ciudadania – 2006 or 2009 version)
  • Ecuadorian Consular Card (Tarjeta De Identification Consular – 2015 version)
  • El Salvadorian Identification Card (Documento Unico de Identidad (DUI) – 2010 version)
  • Guatemalan National Identification Card (Documento Personal de Identificacion (DPI) – 2012 version)
  • Guatemalan Consular Card (Tarjeta de Identificacion Consular – 2002 version)
  • Peruvian Identification Card (Documento Nacional de Identidad (DNI) – 2005 version)
Or you can show another foreign passport: so if you're a dual national then by my reading, you're sorted. Other than that, if you're not from Central/South America and don't have legal residence then you're pretty much sunk. Yay for the major South American nations, except Venezuela or Uruguay, but boo for anyone else.

To recap: if you're an illegal immigrant then you don't really care about driving illegally in the short term. But long term it could be a problem, which is why California has the above AB60 guidance about handing out driving licenses. If you're from Central/South America then they have you covered, otherwise they really don't seem to care. It's perfectly fine for a country to be antagonistic to illegal aliens (that's me struck off Shahid Haque-Hausrath's Christmas card list) but to be arbitrarily receptive to citizens of some countries and not others smacks of, oh I don't know, naked political favouritism?

And now legal immigrants will find it substantially harder to comply with the laws of the state that they're living in - and paying taxes to. Nice one, California.


Ideas that seem attractive but are corporate suicide

A huge loss for popular entertainment when Amazon successfully lured "Top Gear" hosts Clarkson, Hammond and May from the BBC: Apple were trying to hire them too:

Apple is said to have made an unprecedented bid to secure the stars of “Top Gear” when they exited their BBC series earlier this year. But Amazon ended up winning the bidding war for Jeremy Clarkson, James May and Richard Hammond in July.
Can you imagine that? Apple, headquartered in the terminally hip and politically correct city of Cupertino in California, recruiting Mr. Jeremy "Jezza" Clarkson, famous for such quotes as:
  • It's very fast and very, very loud. And then in the corners it will get its tail out more readily than George Michael
  • The problem is that television executives have got it into their heads that if one presenter on a show is a blond-haired, blue-eyed heterosexual boy, the other must be a black Muslim lesbian.
  • Britain's nuclear submarines have been deemed unsafe... probably because they don't have wheel-chair access.
being employed by Apple? Within one week the Apple PR and HR departments would have a "CLARKSON" page, printed on bright red paper bordered with exclamation marks, on the front page of their operational playbooks. The only potential upside for Apple is that Tim Cook, Apple's openly gay CEO, would know with a high degree of certainty what would take up 90% of the allocated time in media interviews, and he's probably got the self assurance to handle it in a relaxed manner - I'm sure he'd rather be asked about Clarkson than about working conditions in Apple's Chinese factories.

It's a crying shame that Amazon, headed by the Dread Pirate Bezos, won the bidding war. When Jezza goes on his next rant to terminally offend half the Western World (and about 0.1% of the rest of the world, who have more pressing and immediate concerns for their welfare than the spoutings of Clarkson), Bezos won't even raise an eyebrow; I can assure you that he doesn't give a bodily functional about the squeals of the masses, as long as Clarkson continues to rake in the dough.


Save the US Postal Service offices!

This is a corker. Today, while wandering past a Staples store in the South Bay I saw a bunch of people outside waving protest signs. Upon closer examination this turned out to be Stop Staples!, a campaign by the American Postal Workers Union (motto "Don't mention Seinfeld"). Staples is the same kind of store in the USA as it is in the UK, providing all kinds of stationery and office supplies. Since late 2013 the US Postal Service has been running a trial program with post office counters in Staples stores, staffed by Staples workers rather than APWU-unionized US Postal Service employees. Hence the protest. The APWU seems rather concerned that the trial program is about to expand.

Reading the AWPU background briefing on the protest is illuminating and amusing in roughly equal measures for anyone who has ever spent time in a US post office:

"Staples and USPS management are perpetrating a fraud on the people of this country," says APWU President Mark Dimondstein. "They are promoting the deal as though taking your mail to Staples is the same as taking it to the Post Office. "It’s not."
He's right, you know. If I go to Staples to make a purchase at the postal counter then I can reasonably expect to be in and out in ten minutes. For the regular post office - once I can find it and get a parking space - I'm budgeting a full hour and bringing a book.
Staples' low-paid, high-turnover employees get just four hours of "classroom" training for postal retail duties.
I don't know about "low-paid". In California they're subject to the state minimum wage which is $9/hour now and $10/hour from January 1st, and there seems to be a thriving demand for competent retail employees. And if the US Postal Service is paying as much as McDonald's for most of their counter staff, they - or rather, the US taxpayer who's funding them - are getting a really bad deal.
Postal workers must pass a test before they are considered qualified to work the window
I can only imagine that it involves the examiner locating a pulse on the worker, with a generous margin for error.

What this is about, of course, is that the APWU is terrified of its membership shrinking, and the associated contributions to the existing retirement plans falling. The USPS retirement plan (healthcare and pensions) funding is in a horrendously bad state as it is, and shrinking the operations, staffing and funding of the USPS will make this situation even clearer, the gap harder to plug, and the public less inclined to back additional federal spending to fill the hole. "Why do I care about the local post office? I go to Staples when I want to post something." The USPS is going to be left with just local letter delivery after Fedex and UPS takes the profitable parcel delivery, and the bulk of those letters are junk mail that the USPS loves for the money and the recipients hate for the spam.

Amusingly, around midday the protestors all left en masse. Presumably they were on their lunch break, a staple feature of US post offices in my experience. As soon as the lunchtime queues start to build up, the counter staff react by closing several of the open counters and wandering off, presumably to have a leisurely lunch. If they've got any eye to the future, I hope they're dusting off their resumés and looking to move to a counter position at Staples before the rush.

Let me quote the APWU leaflet again, in closing:

During the first quarter of Fiscal Year 2014 the USPS enjoyed an operating surplus of $765 million. But the agency’s good news was buried in most media accounts, which said the USPS suffered a loss of $354 million loss. The USPS reported losses for the first quarter of 2014 for one reason – the congressional mandate that requires the Postal Service to pre-fund healthcare benefits for future retirees.
Indeed, damn Congress for insisting that government businesses track their accumulated liabilities as well as their income...


The BBC asks "do people become more prejudiced as they age?"


Psychologists used to believe that greater prejudice among older adults was due to the fact that older people grew up in less egalitarian times. In contrast to this view, we have gathered evidence that normal changes to the brain in late adulthood can lead to greater prejudice among older adults.
There are certainly normal changes to the brain. We call that "life".

Old people have experienced more of life than when they're young, so they have more facts at their disposal to make judgements.

This isn't prejudice. It's postjudice. So the BBC approach of venerating the young and disapproving of the attitudes of older generations is precisely the wrong way around.


The spectacular kind of hardware failure

Gentle reader, I have attempted several times to pen my thoughts on the epic hack of the US Office of Personnel Management that compromised the security information of pretty much everyone who works for the US government, but I keep losing my vision and hearing a ringing in my ears when I try to do so. So I turn to a lesser-known and differently-awesome fail: the US visa system.

Since a computer failure on the 26th of May - over three weeks ago - the US embassies and consulates worldwide have been basically unable to issue new visas except in very limited circumstances. You haven't heard much about this because it hasn't really affected most US citizens, but believe me it's still a big issue. It seems that they're not expecting the system to be working again until next week at the earliest. Estimates of impacted users are on the order of 200,000-500,000; many people are stuck overseas, unable to return to the USA until their visa renewal is processed.

What happened? The US Department of State has a FAQ but it is fairly bland, just referring to "technical problems with our visa systems" and noting "this is a hardware failure, and we are working to restore system functions".

So a hardware failure took out nearly the entire system for a month. The most common cause of this kind of failure is a large storage system - either a mechanical failure that prevents access to all the data you wrote on the disks, or a software error that deleted or overwrote most of the data on there. This, of course, is why we have backups - once you discover the problem, you replace the drive (if broken) and then restore your backed up data from the last known good state. You might then have to apply patches on top to cover data that was written after the backup, but the first step should get you 90%+ of the way there. Of course, this assumes that you have backups and that you are regularly doing test restores to confirm that what you're backing up is still usable.

The alternative failure is of a relatively large machine. If you're running something comparable to the largest databases in the world you're going to be using relatively custom hardware. If it goes "foom", e.g. because its motherboard melts, you're completely stuck until an engineer can come over with the replacement part and fix it. If the part is not replaceable, you're going to have to buy an entirely new machine - and move the old one out, and install the new one, and test it, and hook it up to the existing storage, and run qualification checks... But this should still be on the order of 1 week.

A clue comes from a report of the State Department:

"More than 100 engineers from the government and the private sector [my emphasis] are working around the clock on the problem, said John Kirby, State Department spokesman, at a briefing on Wednesday.
You can't use 100 engineers to replace a piece of hardware. They simply won't fit in your server room. This smells for all the world like a mechanical or software failure affecting a storage system where the data has actually been lost. My money is on backups that weren't actually backing up data, or backing it up in a form that needed substantial manual intervention to restore, e.g. a corrupted database index file which would need every single piece of data to be reindexed. Since they've roped in private sector engineers, they're likely from whoever supplied the hardware in question: Oracle or IBM, at a guess.

The US Visa Office issues around 10 million non-immigrant visas per year, which are fairly simple, and about 500,000 immigrant visas per year which are a lot more involved with photos, other biometrics, large forms and legal papers. Say one of the latter takes up 100MB (a hi-res photo is about 5MB) and one of the former takes up 5MB; then that's a total of about 100TB per year. That's a lot of data to process, particularly if you have to build a verification system from scratch.

I'd love to see a report on this from the Government Accountability Office when the dust settles, but fear that the private sector company concerned will put pressure on to keep the report locked up tight "for reasons of commercial confidentiality and government security". My arse.


Courageous journalism at the BBC

I kid, obviously. When describing the current controversy over the Washington D.C. Metro refusing to take any "issue-oriented" adverts until next year just so that they can avoid showing the prize-winning "Draw Mohammed" cartoon, the BBC resorts to words rather than a picture to describe the salient image.

The advert calls for Americans to support free speech and features a bearded, turban-wearing Muhammad waving a sword and shouting: "You can't draw me!"
In reply, a cartoon bubble portrays an artist grasping a pencil and saying: "That's why I draw you."
How odd, you would have thought that they would have included an image of the cartoon rather than laboriously describe its contents.

Just to make the point, here's the image in question:

The spineless BBC writer isn't shy of displaying their orientation towards issues:

Ms Geller insists the cartoon is a "political opinion" which does not contain any violence.
Ms Geller is of course correct. There's no violence in that picture: the gentleman depicted is holding a sword, but that's as far as it goes. Yet the writer takes particular care to use reported speech and quotes, presumably to demonstrate that he or she is emphatically not in sympathy with Ms Geller or (mysteriously unnamed in the article) artist Bosch Fawstin.

Deary me. Truely, the BBC has resigned from actual journalism in order to be at the back of the line when crocodile feeding time comes around.

I'm really not keen on Pamela Gellar, but the rest of the world seems to be bending over backwards to make her admittedly extreme opinions seem really quite rational and reasonable. And we are surprised when Muslim extremism is emboldened by this obvious cowardice?


Delays are good for you - the MTA proves it

No, really, they do. New York's Metropolitan Transit Authority (something like Transport for London) has produced an outstanding video that shows why making some subway trains late makes others less late:

Yes, the idea is that sometimes delaying a train can prevent further delays by not compounding the gap between trains. Anyone who has waited impatiently on a hot subway platform might find this concept counterintuitive, but transportation experts generally agree that that the evenness of service is as crucial as avoiding individual delays.
The MTA video makes a compelling case. The key insight is that once a platform gets crowded enough, due to constant feed of new passengers and a delayed train, it becomes slower for the next train to debark and embark passengers. So an already delayed train gets more delayed as it progresses down the line. The solution? Spot a train that's getting near the critical delay time and give it priority to progress through the network even if this involves delaying other (less delayed trains).

It's a great example that, even in what we regard as relatively simple systems, there can be a complex interplay between entities that produce highly unintuitive results. Deliberately delaying trains can actually be good for the system as a whole (if not for the passengers sitting in the delayed train with their faces pressed into a fellow passenger's unwashed armpit).


You should care about moving to HTTPS

Eric Mill's "We're Deprecating HTTP and it's going to be okay" is a must-read call-to-arms for everyone with a site on the Internet, explaining why the transition from unencrypted web traffic (HTTP) to encrypted (HTTPS) is actually fundamental to the future existence of the democratic web-as-we-know it.

For the 90% of my reading audience who are already saying "Bored now!" here's why it matters to you. Sir Tim Berners-Lee invented HTTP (the language of communication between web browser and web server) in CERN, a European haven of free thought, trust and international co-operation. The 1930s idea that "Gentlemen do not read each other's mail" was - surprisingly, given the history of cryptographic war in WW2 - fundamental to HTTP; messages might have transited systems owned by several different groups, but none of them would have thought to copy the messages passing through their system, let alone amend them.

This worked fine as long as no-one was interested in the communication of harmless nerds about their hobbies, much as the government-owned Royal Mail doesn't bother to copy the contents of postcards passing through their sorting offices because they only contain inane drivel about sun, sea and sand. However, once people realized that they could communicate freely about their occasionally subversive ideas across borders and continents, and financial institutions woke to the possibility of providing services without paying for expensive un-scalable fallible human cashiers, many governments and other less-legal entities wanted to read (and sometimes alter) Internet traffic.

Mills gives two great examples of where HTTPS prevented - and could have prevented further - nation-state abuse of Internet content:

- The nation of India tried and failed to ban all of GitHub. HTTPS meant they couldn't censor individual pages, and GitHub is too important to India's tech sector for them to ban the whole thing.
- The nation of China weaponized the browsers of users all over the world to attack GitHub for hosting anti-censorship materials (since like India, they can't block only individual pages) by rewriting Baidu's unencrypted JavaScript files in flight.
And closer to home, Cameron's plan to make all online communication subject to monitoring is so stupidly illiberal and expensively pointless that it deserves to be made impractical by general adoption of HTTPS. GCHQ and friends can tap all the Internet traffic they like: if it's protected by HTTPS, the traffic is just taking up disk space to no practical purpose. Brute-forcing, even with nation-state resources, is so expensive that it's reserved for really high-value targets. GCHQ would have to go after something fundamental like a Certificate Authority, which would leave big and obvious fingerprints, or compromise a particular user's machine directly, which doesn't scale.

As long as users are still relaxed about the absence of a padlock in their browser bar, HTTP will continue to provide a route for governments to snoop on their citizens' traffic. So let's give up on HTTP - it has had its day - and move to a world where strongly encrypted traffic is the default.


You can't be too careful - car crashes

The class of systems with high distributed costs and focused but inadequate benefits is going to have another member: auto-calling police in the event of a car crash:

In the event of a crash, the device calls the E.U.'s 911 equivalent (112) and transmits to authorities important information including location, time, and number of passengers in the vehicle. An in-car button will also be installed in all vehicles. The eCall requirement will add an estimated $100 to the price of a car.
$100 on each (new) car sold: so how many new cars are sold in the EU each year? About 14 million in 2012. So this measure will cost $1.4 billion, and maybe $150 million in the UK. What's the benefit?
Each year nearly 26,000 people are killed in the E.U. by car crashes. This new device is estimated to reduce that number by 10 percent, saving 2,600 lives annually, by cutting down emergency response time by as much as 60 percent.
The cost of a life for purposes of safety varies by country and mode of transport, but let's take $1 million as the average. Given the quoted statistics, $2.6 billion saving (though optimistic, probably lower) comprehensively dwarfs $1.4 billion cost (though also optimistic, probably higher). Why isn't this a slam-dunk decision?

The problem is twofold: a) zeroing cost for lives saved, and b) the assumption of 10% saving. Let's consider each in turn.

If an injury is potentially fatal but not actually fatal due to timely intervention, it's almost always due to either early suppression of severe blood loss, or timely (within 1-2 mins) clearing of obstructed airway. The latter isn't relevant due to emergency service response times, so we only consider the former. This injured person will still need emergency treatment followed by several days of hospital care, and quite possibly follow-on care of injuries, rehab, and in some cases reduced lifetime tax payments due to reduced earnings and disability payments, so you're looking at order of $100K average costs. That's still not really significant.

However, consider a typical case where a life is saved: a car driver has an accident in the countryside when no-one is around. His car calls 112 and so the police (not the ambulance service initially, because they are too stretched to respond to wild goose chases) respond to his location. Seeing the crash they call for an ambulance which arrives 10-30 minutes before it would have otherwise arrived due to a passer-by report - people tend to notice a crashed car with no emergency services around it. He would have died due to shock (depletion of oxygen to the critical organs due to blood loss / asphyxiation / traumatic damage to heart and lungs) but the ambulance got there in time to oxygenate him and transport to hospital. Just how common is this?

Fatal road accidents rarely happen on remote roads - unsurprisingly, they happen where there are many more cars and roadside obstructions to run into. If an accident happens where passers-by are prevalent, this system doesn't help at all since nearly all passers-by have mobile phones. So we're only looking at a small fraction - 5% is optimistic - of accidents. The press release assumed 10%, so the benefit has already halved and is perilously close to the cost.

But bleeding to death is not a common cause of death from road accidents for drivers/passengers. Much more likely is traumatic head injury, which tends to kill them right there in the car. Unsecured drivers/passengers fly through the windscreen, or secured drivers/passengers bang their head against the car frame. This kills instantly, or in a few minutes. Another mechanism is the "third collision" where the car bangs into a tree (collision 1), the driver bangs into their seatbelt (collision 2) and then the free-hanging organs like lungs, heart bang into the drivers chest, or their blood vessels bang into ligaments that cheesewire them (collision 3). If you're in this situation and your aorta (the major blood vessel coming out of the heart) is damaged you can expect a 60%-80% chance of death no matter how quickly you get to the hospital.

Therefore, before we stick the European population with an extra $1 billion of annual costs, why don't we conduct a limited experiment introducing this requirement into a single country which is similar to another country in road crash death rates to see what effect, if measurable, this measure has? Or is the notion of trade-offs too alien to the EU?