2014-10-22

State-endorsed web browsers turn out to be bad news

Making the headlines in the tech world this week has been evidence of someone trying to man-in-the-middle Chinese iCloud users:

Unlike the recent attack on Google, this attack is nationwide and coincides with the launch today in China of the newest iPhone. While the attacks on Google and Yahoo enabled the authorities to snoop on what information Chinese were accessing on those two platforms, the Apple attack is different. If users ignored the security warning and clicked through to the Apple site and entered their username and password, this information has now been compromised by the Chinese authorities. Many Apple customers use iCloud to store their personal information, including iMessages, photos and contacts. This may also somehow be related again to images and videos of the Hong Kong protests being shared on the mainland.
MITM attacks are not a new phenomenon in China but this one is widespread, and clearly needs substantial resources and access to be effective. As such, it would require at least government complicity to organise and implement.

Of course, modern browsers are designed to avoid exactly this problem. This is why the Western world devotes so much effort to implementing and preserving the integrity of the "certificate chain" in SSL - you know you're connecting to your bank because the certificate is signed by your bank, and the bank's signature is signed by a certificate authority, and your browser already knows what the certificate authority's signature looks like. But it seems that in China a lot of people use Qihoo 360 web browser. It claims to provide anti-virus and malware protection, but for the past 18 months questions have been asked about its SSL implementation:

If your browser is either 360 Safe Browser or Internet Explorer 6, which together make up for about half of all browsers used in China, all you need to do is to click continue once. You will see no subsequent warnings. 360's so-called "Safe Browser" even shows a green check suggesting that the website is safe, once you’ve approved the initial warning message.

I should note, for the sake of clarity, that both the 2013 and the current MITM reports come from greatfire.org, whose owners leave little doubt that they have concerns about the current regime in China. A proper assessment of Qihoo's 360 browser would require it to be downloaded on a sacrificial PC and used to check out websites with known problems in their SSL certificates (e.g. self-signed, out of date, being MITM'd). For extra points you'd download it from a Chinese IP. I don't have the time or spare machine to test this thoroughly, but if anyone does then I'd be interested in the results.

Anyway, if the browser compromise checks out then I'm really not surprised at this development. In fact I'm surprised it hasn't happened earlier, and wonder if there have been parallel efforts at compromising IE/Firefox/Opera/Chrome downloads in China: it would take substantial resources to modify a browser installer to download and apply a binary patch to the downloaded binary which allowed an additional fake certificate authority (e.g. the Chinese government could pretend to be Apple), and more resources to keep up to date with browser releases so that you could auto-build the patch shortly after each new browser version release, but it's at least conceivable. But if you have lots of users of a browser developed by a firm within China, compromising that browser and its users is almost as good and much, much easier.

2014-10-13

Corporate welfare from Steelie Neelie and the EU

I used to be the starry-eyed person who thought that governments pouring into a new concept for "research" was a good thing. That didn't last long. Now I read The Reg on the EU's plan to chuck 2.5 billion euros at "Big Data" "research" and wonder why, in an age of austerity, the EU thinks that pissing away the entire annual defence budget of Austria is a good idea.

First, a primer for anyone unfamiliar with "Big Data". It's a horrendously vague term, as you'd expect. The EU defines the term thus:

Big data is often defined as any data set that cannot be handled using today’s widely available mainstream solutions, techniques, and technologies.
Ah, "mainstream". What does this actually mean? It's a reasonable lower bound to start with what's feasible on a local area network. If you have a data set with low hundreds of terabytes of storage, you can store and process this on some tens of regular PCs; if you go up to about 1PB (petabyte == 1024 terabytes, 1 terabyte is the storage of a regular PC hard drive) then you're starting to go beyond what you can store and process locally, and need to think about someone else hosting your storage and compute facility.

Here's an example. Suppose you have a collection of overhead imagery of the United Kingdom, in the infra-red spectrum, sampled at 1m resolution. Given that the UK land area is just under 250 thousand square kilometers, if you represent this in an image with 256 levels of intensity (1 byte per pixel) you'll need 250,0000 x (1000 x 1000) = 250 000 000 000 pixels or 250 gigabytes of storage. This will comfortably fit on a single hard drive. If you reduce this to 10cm resolution - so that at maximum resolution your laptop screen of 1200 pixel width will show 120m of land - then you're looking at 25 TB of data, so you'll need a network of tens of PCs to store and process it. If, instead of a single infra-red channel, you have 40 channels of different electromagnetic frequencies, from low infra-red up to ultra violet, you're at 1PB and need Big Data to solve the problem of processing the data.

Another example, more privacy-concerning: if you have 1KB of data about each of the 7bn people in the world (say, their daily physical location over 1 year inferred from their mobile phone logs), you'll have 7 terabytes of information. If you have 120 KB of data (say, their physical location every 10 minutes) then this is around 1PB and approaches the Big Data limits.

Here's the press release:

Mastering big data could mean:
  • up to 30% of the global data market for European suppliers;
  • 100,000 new data-related jobs in Europe by 2020;
  • 10% lower energy consumption, better health-care outcomes and more productive industrial machinery.
My arse, but let's look at each claim in turn.
  • How is this project going to make it more likely for European suppliers to take over more of the market? Won't all the results of the research be public? How, then, will a European company be better placed to take advantage of them than a US company? Unless one or more US-based international company has promised to attribute a good chunk of its future Big Data work to its European operations as an informal quid-pro-quo for funding from this pot.
  • As Tim Worstall is fond of saying, jobs are a cost not a benefit. These need to be new jobs that are a prerequisite for larger Big Data economic gains to be realized, not busywork to meet artificial Big Data goals
  • [citation required] to quote Wikipedia. I'll believe it when I see it measured by someone without financial interest in the Big Data project.

The EU even has a website devoted to the topic: Big Data Value. Some idea of the boondoggle level of this project can be gleaned from the stated commitment:

... to build a data-driven economy across Europe, mastering the generation of value from Big Data and creating a significant competitive advantage for European industry, boosting economic growth and jobs. The BDV PPP will commence in 2015[,] start with first projects in 2016 and will run until 2020. Covering the multidimensional character of Big Data, the PPP activities will address technology and applications development, business model discovery, ecosystem validation, skills profiling, regulatory and IPR environment and social aspects.
So how will we know if these 2.5bn Euros have been well spent? Um. Well. Ah. There are no deliverables specified, no ways that we can check back in 2020 to see if the project was successful. We can't even check in 2017 whether we're making the required progress, other than verifying that the budget is being spent at the appropriate velocity - and believe me, it will be.

The fundamental problem with widespread adoption of Big Data is that you need to accumulate the data before you can start to process it. It's surprisingly hard to do this - there really isn't that much new data generated in most fields and you can do an awful lot if you have reasonably-specced PCs on a high-speed LAN. Give each PC a few TB in storage, stripe your data over PCs for redundancy (not vulnerable to failure of a single drive or PC) and speed, and you're good to go. Even if you have a huge pile of storage, if you don't have the corresponding processing power then you're screwed and you'll have to figure out a way of copying all the data into Amazon/Google/Azure to allow them to process it.

Images and video are probably the most ripe field for Big Data, but still you can't avoid the storage/processing problem. If you already have the data in a cloud storage provider like Amazon/Google/Azure, they likely already have the processing models for your data needs; if you don't, where are all the CPUs you need for your processing? It's likely that the major limitations processing Big Data in most companies is appropriate reduction of the data to a relatively small secondary data set (e.g. processing raw images into vectors via edge detection) before sending it somewhere for processing.

The EU is about to hand a couple billion euros to favoured European companies and university research departments, and it's going to get nine tenths of squat all out of it. Mark my words, and check back in 2020 to see what this project has produced to benefit anyone other than its participants.

2014-09-25

Signs that the terrorism threat might be overblown

Or maybe just a sign that the US education system is a pool of sharks...

Modern terrorism getting you down? Don't worry, it's an opportunity for you! Sign up for a certificate in Terrorism Studies!

In the program, you will develop an understanding of terrorism and counter-terrorism. The online program is suitable for students interested in pursuing a career in homeland security at local, state, or federal levels; joining national and international counter-terrorism agencies; conducting research on terrorism in academia; or seeking opportunities in relevant industries.
Presumably it's also suitable for students interested in pursuing a career in terrorism? Or maybe this is an elaborate honey trap by the FBI, but I suspect that a) they don't have the motivation and b) they can't afford to fund the course.

2014-09-19

Don't ask for your emails to be deleted

Darrell Issa, Republican congressman from California (yes, amazingly they exist) releases the oversight report on the initial rollout of Healthcare.gov and it wasn't pretty. The bulk of the report was based off emails that they managed to retrieve from Health + Human Services and their CMS subsidiary, and the report authors did a nice job of excerpting the damning snippets from the emails that confirmed everyone's suspicions about the rollout: the grunts implementing and testing the site knew darned well that it wasn't ready, but they were overridden.

I don't find any particular reason in the report to believe that the President knew the site wasn't ready; it looks very much like he and his advisors were assured that everything was in hand, and he had no particular reason to disbelieve it. The problems occurred lower down in the hierarchy:

Mr. Sivak showed Mr. Baitman emails that were made public by Congress in the wake of Healthcare.gov's disastrous launch. In these emails, dated September 27, 2013 [launch date was Oct 1st], a CMS official working on the FFM development, wrote "the facts are that we have not successfully handled more than 500 concurrent users filling out applications in an environment that is similarly in size to Day 1 production." In response, Mr. Baitman wrote "Frankly, it’s worse than I imagined!" Mr. Sivak replied, "Anyone who has any software experience at all would read that and immediately ask what the fuck you were thinking by launching."
Indeed, we were asking almost exactly that question. And there was no naivety about motivations:
How did one week Henry Chao tell us there was no way Account Transfer would be ready, then a meeting at the White House and a week later, oh, yeah, everything is back on track, we’ll meet the dates? That’s what I mean by WTF. You could definitely see the CYA moves coming a mile away
Doublethink is clearly very important for project managers. Henry Chao was one of the prime Healthcare.gov project managers and it appears he knew that the site was heading to disaster, but for some reason he couldn't or wouldn't articulate this to the administration.

Issa, of course, has plenty of partisan reasons to bash the administration and the Healthcare.gov backers, but it's hard to conclude anything other than that this launch was destined to crash and burn spectacularly, that this was known well in advance, and that it was egregiously mis-managed. That Mikey Dickerson and his crew managed to retrieve some semblance of success from this state was amazing, but not something that should be relied on by any future project manager.

Once again, the maxim "Do not write anything in an email that you do not want to see on the front page of a major newspaper" is confirmed. The usual wisdom around this is a combination of a) mail is transferred in the clear between servers on the public internet, although this is changing, and b) the risk of including the wrong person on your To: or Cc: lines. This report highlights a third option: the risk that your email will be retrieved during a legal discovery process. If you send your email from a company email system it'll be archived there and prone to later legal discovery even if you and the recipient delete it. This also applies if any of your recipients use a company or government email address.

The Verge provides a nice summary of the highlights in the report if you don't have the stomach to read the whole thing.

2014-09-08

Take the upside and you own the downside

I was annoyed by this inane Reuters article on the fate of the UK's gold stash:

An independent Scotland could lay claim to a part of the United Kingdom's 310-tonne gold reserves if votes go in favour of the "Yes" campaign this month, with ownership of Britain's bullion hoard up for negotiation along with other assets.
If I were Scotland, I'd run as far as possible from the £7.8bn pile of gold bricks. The reason I'd do this is because if I take on a fraction of the assets of the UK, I have no argument against also taking on its liabilities:
As of Q1 2013 UK government debt amounted to £1,377 billion, or 88.1% of total GDP, at which time the annual cost of servicing the public debt amounted to around £43bn, or roughly 3% of GDP.
Why would you take (say) 10% of £7.8bn when you'd also have to assume 10% of a £1400bn liability? You'd have to be stark staring bonkers. Alex Salmond isn't a rocket scientist, but even he would realise how dumb this would be.

2014-09-06

New clamping down on information in China

Spotted this on a net security research blog yesterday: someone is trying to snoop on the web traffic of Chinese students and researchers:

All evidence indicates that a MITM [man-in-the-middle] attack is being conducted against traffic between China’s nationwide education and research network CERNET and www.google.com. It looks as if the MITM is carried out on a network belonging to AS23911, which is the outer part of CERNET that peers with all external networks. This network is located in China, so we can conclude that the MITM was being done within the country.
To decipher this, readers should note that CERNET is the Chinese network for education and research - universities and the like. The regular Great Firewall of China blocking is fairly crude and makes it practically difficult for researchers to get access to the information they need, so CERNET users have mostly free access to the Internet at large - I'm sure their universities block access to dodgy sites, but to be fair so do Western universities. What's happening is that someone is intercepting - not just snooping on - their requests to go to www.google.com and is trying to pretend to be Google.

The reason the intercept is failing is because Google - like Facebook, Yahoo, Twitter and other sites - redirects plain HTTP requests to its homepage to a HTTPS address, so most people bookmark those sites with an HTTPS address. Therefore the users were requesting https://www.google.com/ and the attackers had to fake Google's SSL certificate. Because of of the way SSL is designed, this is quite hard; they couldn't get a reputable Certificate Authority to sign their certificate saying "sure, this is Google" so they signed it themselves, much like a schoolchild signing a note purportedly from their parent but with their own name. Modern browsers (Chrome, Firefox, modern versions of IE) warn you when this is happening, which is how the users noticed. The Netresec team's analysis showed that the timings of the steps of the connection indicated strongly that the interceptor was somewhere within China.

The attack doesn't seem to be very sophisticated, but it does require reasonable resources and access to networking systems - you've got to reprogram routers in the path of the traffic to redirect the traffic going to Google to come to your own server instead, so you either need to own the routers to start with or compromise the routers of an organisation like a university. Generally, the further you get from the user you're intercepting, the greater your resources need to be. It would be interesting to know what fraction of traffic is being intercepted - the more users you're intercepting, the more computing resource you need to perform the attack because you've got to intercept the connection, log it, and then connect to Google/Twitter/Yahoo yourself to get the results the user is asking for.

The attempted intercepts were originally reported on the Greatfire.org blog which observes that there were several reports from around CERNET of this happening. Was this a trial run? If so it has rather blown up in the faces of the attackers; now the word will circulate about the eavesdropping and CERNET users will be more cautious when faced with odd connection errors.

If the attackers want to press on, I'd expect the next step to be more sophisticated. One approach would be SSL stripping where the interceptor tries to downgrade the connection - the user requests https://www.twitter.com/ but the attacker rewrites that request to be http://www.twitter.com/. The user's browser sees a response for http instead of https and continues with an unencrypted connection. Luckily, with Twitter this will not work well. If you run "curl -I https://www.twitter.com/" from a command line, you'll see this:

HTTP/1.1 301 Moved Permanently
content-length: 0
date: Sat, 06 Sep 2014 17:23:21 UTC
location: https://twitter.com/
server: tsa_a
set-cookie: guest_id=XXXXXXXXXXXXXXXXX; Domain=.twitter.com; Path=/; Expires=Mon, 05-Sep-2016 17:23:21 UTC
strict-transport-security: max-age=631138519
x-connection-hash: aaaaaaaaaaaaaaaa
That "strict-transport-security" line tells the browser that future connections to this site for the next N seconds must use HTTPS, and the browser should not continue the connection if the site tries to use HTTP. This is HTTP Strict Transport Security (HSTS) and Twitter is one of the first big sites I've seen using it - Google and Facebook haven't adopted it yet, at least for their main sites.

Alternatively the interceptor may try to compromise a reputable certificate authority so it can forge SSL certificates that browsers will actually accept. This would be a really big investment, almost certainly requiring nation-state-level resources, and would probably not be done just to snoop on researchers - if you can do this, it's very valuable for all sorts of access. It also won't work for the major sites as browsers like Chrome and Firefox use certificate pinning - they know what the current version of those sites' SSL certs look like, and will complain loudly if they see something different.

The most effective approach, for what it's worth, is to put logging software on all the computers connected to CERNET, but that's probably logistically infeasible - it only works for targeting a small number of users.

So someone with significant resources in China is trying to find out what their researchers are searching for. Is the government getting nervous about what information is flowing into China via this route?

2014-09-03

Surrender monkeys don't eat balut

A fascinating shit-storm is brewing between the Philippine Army and the UN Disengagement Observer Force as a result of recent events in the Golan Heights:

The Philippine military said Monday that a U.N. peacekeeping commander in the Golan Heights should be investigated for allegedly asking Filipino troops to surrender to Syrian rebels who had attacked and surrounded their camp.
[...]
When the besieged Filipino troops sought his [Gen. Catapang's] advice after they were ordered to lay down their arms as part of an arrangement with the rebels to secure the Fijians' release, Catapang said he asked them to defy the order.
It seems that in order to facilitate negotiations for the release of 45 Fijian soldiers captured by the (al-Qaeda affiliated) Nusra Front rebels - such capture perhaps due to less-than-stellar planning by UNDOF - the UNDOF commander decided that yielding to the rebels' demands for the Filipino troops to give up their weapons would be just dandy. After all, what could possibly go wrong?

Gen. Catapang is Chief of Staff of the Philippine Armed Forces, so can't really rise any higher in the command structure, and isn't well-known enough to run for high government office, so he's got no real motive to puff up his role in this dispute. I'm inclined to believe the main thrust of his account. Since the army has been in near-continuous counter-insurgency campaigns, with the communist NPA in the central Philippines and the Islamic groups in the south and south west, they've accumulated quite a lot of experience with fanatic groups and have presumably absorbed the lesson that doing what your opponent tells you to seldom works out well.

It'll be interesting to see if the resolution of the dispute is made public:

Catapang said an investigation would allow the UNDOF commander to explain his side and the Philippine military to explain why it advised the Filipino peacekeepers to defy his order.
I doubt the second part will take very long. I'd start with "Because it was bloody stupid" and work up from there. Catapang, as a 4-star general, comfortably out-ranks UNDOF's 2-star leader and so there's no insubordination problem I can see. The first part would be educational though: just what did the UNDOF commander think would happen if the Filipino troops had laid down their arms as ordered? And what involvement did the UNDOF commander have in the Fijians being captured in the first place? The Philippine Army is withdrawing from the UNDOF mission in the Golan, presumably because they have no appetite for being put in the same position again when UNDOF decides that covering its backside is more important than the safety of the troops in its command.

It seems that si vis pacem, para bellum is still true: if you want to keep the peace, you have to be prepared to kick the ass.

Update: Richard Fernandez at the Belmont Club is well worth reading on this topic:

In the past the UN apparatchiks have relied on the faithfulness of their subordinate commanders to take a bullet for the team. "Theirs not to reason why, theirs but to do and die." But Tennyson had never been to the Philippines where the word for blindly following orders is tanga – or sap.

2014-08-13

A voice of reason in CiF

It would have to be a mathmo, wouldn't it? Sam Howison, an applied maths professor, looks at why the first 50 Fields medal winners were uniformly male and, refreshingly, comes up with a range of explanations with the starting point that there just aren't many female mathmos:

Data is scarce in this rarefied region, and hypotheses are hard to test; so, too, is the influence of the culture of their chosen field. Nevertheless, such astronomical odds of a woman winning the medal are disturbing, and they are just an extreme point of a range of evidence that women are underrepresented in mathematics at many levels.
It's indisputably true that you don't find anything like a 50% proportion of women at the top level of maths, or theoretical computer science for that matter. On the other hand, in my experience the women that you do find there aren't obviously any less smart and capable than the men, so if you were making randomized choices based on intellect you'd expect women to be far more frequent in Fields medal holders than they are.

This year, Stanford professor Maryam Mirzakhani won a Fields medal. She's clearly a hard-core pure mathmo; I defy anyone with anything less than a Ph.D. in maths to read about her research interests and not have their brain leak out of their ears. This is not just "I don't understand what this is about", this is "I can't even picture the most basic explanation of this in my head". Compared to that, even Fermat's Last Theorem was a walk in the park - solving polynomial equations is standard A-level fare, and even if you can't understand what Andrew Wiles did to prove it you can at least understand the problem. With Mirzakhani's work, you have no frame of reference, you're like a child who wanders into the middle of a movie.

Howison's point about the astronomical odds of the Fields medal award gender distribution (50 tails in 51 unbiased coin tosses) is a nice point of probability, but of course the first place you'd start is to look at the eligible pool - top-flight mathematicians, generally at (UK) professor level, with a substantial track record of publishing. That will tell you your bias; if 1 in 10 people in the pool are female, you're tossing a biased coin which will show tails 9 times out of 10. Still, it's pretty clear that even with that pool the Fields medal gender split is way out of line with what you'd expect.

Howison makes an interesting point that I hadn't considered up to now:

[...] people with successful careers have usually had a high degree of support from a mentor. As well as providing academic guidance and inspiration (as Mirzakhani freely acknowledges she had when a student), the mentor will introduce their charge to influential colleagues on the conference circuit and elsewhere, and arrange invitations to speak at seminars and workshops. That is one way for a young mathematician to get their work noticed, and to improve their chances of getting a position in a world-leading department where they can thrive. Is this perhaps (if only subconsciously) difficult for women in a community where the majority are men?
The usual reason for explaining the lack of women in senior positions in Fortune 500 firms (banks, Big Pharma etc.) is that they're not as good at men at talking their own book, preferring to be more even-handed in giving credit for the achievements in which they'd participated. However, Howison tantalisingly hints at a squaring function in gender representation here - will junior female mathmos only get good support and PR from a senior female mentor, and do such senior female mathmos pick up juniors with a blind eye to gender? It would be fascinating to get some data here.

I do wonder whether that perennial topic in gender discrimination, motherhood, plays a role here. Because the Fields medal only goes to people younger than 40 - Andrew Wiles, who cracked Fermat's Last Theorem, was a notable omission from its holders due to his age - if you take time out from academe to have children then this disproportionately affects your time where you're eligible for a Fields medal. The Guardian interviewed this year's sole female awardee, Maryam Mirzakhani but she didn't make any comment about her family life so I have no idea if she has kids.

So mad props to Maryam Mirzakhani for being the first female winner of the Fields medal, and here's to hoping for many more. Apart from anything else, if we can start to get some data on what factors determine female Fields medal winners we might have a hazy glimpse of what we need to fix in the academic lifecycle to get more top-flight women choosing to follow it.

2014-08-11

Formalising success in a bureaucracy

It's only natural, when you've managed to get out of a hole against all odds, that you want to re-use the people and/or planning that made the difference. You'd be wasteful if you didn't, to be honest. Following this line of thinking, and after a small team of digital fixers managed to save the flagship Healthcare.Gov federal healthcare exchange from near-certain doom, the White House is trying to do just that.

Today they announced the launch of the new U.S. Digital Service which aims to replicates the lessons of the (relative) success in saving Healthcare.Gov with other troubled US federal government IT projects. Heaven knows that there's no shortage of potential targets for USDS to help with. The question of the moment is: can this new government team actually succeed? If so, what does success look like?

US CIO Steve van Roekel outlined the USDS role:

"This isn't going to be a group that we parachute in to write code," as Van Roekel put it in a call earlier this summer, and with perhaps the Department of Health and Human's experience with HealthCare.gov on the brain, "This isn't decending a group of developers onto the scene." Rather, the focus is going to be on helping agencies figure out where their weak points are and how to fix them.
Note that therefore the role of USDS staff isn't actually the same as the Healthcare.Gov fixers, but that might be OK as the fixing itself wouldn't scale; if you want to solve the key IT problems of more than one government agency at at time then you can't have most your staff embedded in one project, and there's no reason to think that the government can recruit multiples of the motivated team that fixed Healthcare.gov. They're going to have to strike a balance, though. They won't be able to determine the principal IT problems of an agency without spending time working with and talking to the agency's tech team. The more time they spend there, the more trust they'll gain and the better the quality of information they'll gather - but then they won't be able to help as many agencies.

The danger with any new government agency is that after a time it accumulates bureaucrats whose primary purpose is propagating their own employment and importance. Van Roekel seems to be aware of this and planning to bring in people for 2-4 year rotations. With placements of 3-6 months this may be about right; long enough for the new people to spend a placement or two with the veterans and absorb the institutional knowledge, do a couple more placements as peers while encouraging their friends to join up, then lead new recruits in placements as the veterans leave.

What's going to be interesting is to see how the USDS embeds are treated in the troubled agencies. Are they going to have the influence and effective power to remove obstructions - such as long-term barnacle workers who hoard knowledge and obstruct progress? If not, they're unlikely to be able to change much. If so, the agency's workers are going to hunker down and be terrified of being fired or reassigned. It's going to be quite a challenge for tech sector workers to get their heads around the government worker mindset sufficiently to influence those workers into getting things fixed.

Incidentally, www.usds.gov was not resolving as of posting time; I actually consider that a potential sign of success as the new team is focusing on getting operational before getting any marketing/PR in place; still, they're going to need a portfolio of some form after a few months in order to attract their new short-term hires.

2014-07-30

Bringing the diversity of car manufacturers to Silicon Valley

I should start this blog by warning the reader of my prejudice towards Jesse Jackson. I think he's a fairly despicable human being; a race hustler who is standing on the shoulders of the giants of the US Civil Rights Movement (Parks, MLK et al) to further his own petty shakedown rackets and attempts to gain political power.

That said, let's examine his latest crusade: bringing the focus of the US Equal Employment Opportunity Commission onto the diversity disaster area that is Silicon Valley.

"The government has a role to play" in ensuring that women and minorities are fairly represented in the tech workforce, Jackson told a USA TODAY editorial board meeting. He said the U.S. Equal Employment Opportunity Commission needs to examine Silicon Valley's employment contracts.
The trigger for this appears to be Twitter's release of workforce diversity statistics (select the Twitter tab, the default is Yahoo). They show a global 70% male workforce with 50% white, 29% Asian, 3% Hispanic, 2% black, 3% mixed and 4% other. Jackson claims that this is proof that the EEOC needs to step in. Because what could possibly go wrong with that?

The gaping hole in USA Today's argument:

Of Twitter's U.S. employees, only 3% are Hispanic and 5% black, but those groups along with Asian Americans account for 41% of its U.S. users.
Wow, talk about a misleading stat. I assume "mixed" is rolled in with "black" to make the 5%, using the Halle Berry "one drop of blood" theory, but note that if you add Asian Americans in it becomes:
Of Twitter's U.S. employees, only 3% are Hispanic and 5% black plus 29% Asian making 37% total, but those groups account for 41% of its U.S. users.
Hmm, that's a little bit different, no?

Since Silicon Valley is in focus, let's look at the demographics in the Bay Area from the 2010 census:

  • 52.5% White including white Hispanic
  • 6.7% non-Hispanic African American
  • 23.3% Asian (7.9% Chinese, 5.1% Filipino, 3.3% Indian, 2.5% Vietnamese, 1.0% Korean, 0.9% Japanese plus rounding errors for others)
  • 23.5% Hispanic or Latino of any race (17.9% Mexican, 1.3% Salvadoran)
  • 5.4% from two or more races
  • 10.8% from "other race"
The categories aren't an exact overlap, but you'll note that whites are almost exactly represented in Twitter as in the Bay Area population. Asians are over-represented in Twitter (29% vs 23%), African Americans under-represented (7% vs 5%) but the real under-representation is Hispanic (24% vs 3%). Why is that? Hispanics in California are disproportionately over-represented in the menial jobs currently. This is starting to change a little with the new generation of America-born Hispanic kids but their parents can't generally afford top-tier universities for engineering or CS courses so it'll be at least one more generation before they start to appear in the engineering/CS student pool for recruitment.

The really disgusting thing about Jackson is when you realize what he is actually implying - that Silicon Valley engineers systematically discriminate in hiring against black and Hispanic engineers just on the basis of their skin colour. Yet somehow they discriminate in favour of Chinese and Indian engineers on the same basis - so they're racist, but very narrowly so. What Jackson fails to point out - because it wrecks his entire thesis - is that the real demographic problem is in the pool of engineers eligible for these jobs. African-American and Hispanic students are massively under-represented here. This isn't Twitter's fault, or Google's fault, or Facebook, Apple, or IBM. The problem starts at the awful public (state) schools which poor American students attend and which completely fail to give them any reasonable preparation for university courses with objective (numeric) subjects - maths, computer science, physics - that are the grounding for computer science careers. But delving into those facts might take an enquiry into unionised teaching and teacher tenure rules, and I'd bet Jesse's union buddies wouldn't like that.

The engineers I know who conduct interviews for computing firms day in, day out, are overwhelmingly thoughtful and fair individuals who strive to give any new candidate a fair go at getting hired. Even the occasional monster among them is uniformly brutal - white, Chinese and Indian candidates have as brutually intellectual an interview as Hispanic and black candidates. If Jackson were to appear before those engineers and accuse them explicitly of bad-faith prejudice against black and Hispanic candidates, they'd probably punch him.

The real problem in Silicon Valley demographics is the male vs female disparity in engineering. There are plenty of good, smart, talented women - they're just not going into engineering. Until we figure out why, we're missing out on a heck of a lot of talent. But Jackson is not pushing this angle - perhaps he's figured out that he has nothing to say on the subject and so there's no money in it for him and his cronies.

I can do no better than conclude with Jackson's own words:

The former two-time Democratic presidential candidate said he'll continue pushing the issue and has no plans to retire. "The struggle for emancipation is my life," he said in an interview. "It's my calling."
Well it's your revenue stream, at least. God, that man gets on my wick.