DDoS and the Tragedy of the Commons of the Internet of Things

On Friday there was a massive Distributed Denial of Service attack on DynDNS, who provide Domain Name services to a number of major companies including Twitter, Spotify and SoundCloud, effectively knocking those sites offline for a significant fraction of the global population. Brian Krebs provides a useful summary of the attack; he is unusually well versed in these matters because his website "Krebs on Security" was taken offline on 20th September after a massive Internet-of-Things-sourced DDoS against it. It seems that Krebs' ongoing coverage and analysis of DDoS with a focus on the Internet of Things (IoT) - "smart" Internet connected home devices such as babycams and security monitors - raised the ire of those using the IoT for their nefarious purposes. It proved necessary to stick Krebs' blog behind Google's Project Shield which protects major targets of information suppression behind something resembling +5 enchanted DDoS armour.

Where did this threat to the Internet come from? Should we be worried? What can we do? And why is this whole situation a Tragedy of the Commons?

Primer on DNS

Let's look at Friday's outage first. Dyn DNS is a DNS hosting company. They provide an easy way for companies who want a worldwide web presence to distribute information about the addresses of their servers - in pre-Internet terms, they're like a business phone directory. Your company Cat Grooming Inc., which has bought the domain name catgrooming.com, has set up its web servers on Internet addresses and, and its mail server on Somehow, when someone types "catgrooming.com" in their internet brower, they need that translating to the right numerical Internet address. For that translation, their browser consults the local Domain Name Service (DNS) server, which might be from their local ISP, or a public one like Google's Public DNS ( and

So if Cat Grooming wants to change the Internet address of their webservers, they either have to tell every single DNS server of the new address (impractical), or run a special service that every DNS server consults to discover up to date information for the hostnames. Running a dedicated service is expensive, so many companies use a third party to run this dedicated service. Dyn DNS is one such company: you tell them whenever you make an address change, and they update their records, and your domain's information says that Dyn DNS does its address resolution.

To check whether a hostname on the web uses DynDNS, you can use the "dig" command which should work from the Linux, MacOS or FreeBSD command line:

$ dig +short -t NS twitter.com
This shows that twitter.com is using Dyn DNS because it has dynect.net hostnames as its name servers.

Your browser doesn't query Dyn DNS for every twitter.com URL you type. Each result you get back from DNS comes with a "time to live" (TTL) which specifies for how many seconds the answer is valid. If your twitter.com query came back as with a TTL of 3600 then your browser would use that address for the next hour without bothering to check Dyn DNS. Only after 1 hour (3600 seconds) would it re-check Dyn DNS for an update.

Attack mechanism

The Internet of Things includes devices such as "babycams" which enable neurotic parents to keep an eye on their child's activities from elsewhere in the house, or even from the restaurant to which they have sneaked out for a couple of hours of eating that does not involve thrown or barfed food. The easiest way to make these devices accessible from the public Internet is to give them their own Internet address, so you can enter that address on a mobile phone or whatever and connect to the device. Of course, the device will challenge any new connection attempt for a username and password; however, many devices have extremely stupid default passwords and most users won't bother to change them.

Over the past decade, Internet criminals have become very good at scanning large swathes of the Internet to find devices with certain characteristics - unpatched Windows 2000 machines, webcams, SQL servers etc. That lets them find candidate IoT devices on which they can focus automated break-in attempts. If you can get past the password protection for these devices, you can generally make them do anything you want. The typical approach is to add code that makes them periodically query a central command-and-control server for instructions; those instructions might be "hit this service with queries randomly selected from this list, at a rate of one query every 1-2 seconds, for the next 4 hours."

The real problem with this kind of attack is that it's very hard to fix. You have to change each individual device to block out the attackers - there's generally no way to force a reset of passwords to all devices from a given manufacturer. The manufacturer has no real incentive to do this since it has the customer's money already and isn't obviously legally liable for the behavior. The owner has no real incentive to do this because this device compromise doesn't normally materially affect the device operation. You can try to sell the benefits of a password fix - "random strangers on the internet can see your baby!" but even then the technical steps to fix a password may be too tedious or poorly explained for the owner to action. ISPs might be able to detect compromised devices by their network traffic patterns and notify their owners, but if they chase them to fix the devices too aggressively then they might piss off the owners enough to move to a different ISP.

Why don't ISPs pre-emptively fix devices if they find compromised devices on their network? Generally, because they have no safe harbour for this remedial work - they could be prosecuted for illegal access to devices. They might survive in court after spending lots of money on lawyers, but why take the risk?

Effects of the attack

Dyn DNS was effectively knocked off the Internet for many hours. Any website using Dyn DNS for their name servers saw incoming traffic drop off as users' cached addresses from DNS expired and their browsers insisted on getting an up-to-date address - which was not available, because the Dyn DNS servers were melting.

Basic remediation for sites in this situation is to increase the Time-to-Live setting on their DNS records. If Cat Grooming Inc's previous setting was 3600 seconds, then after 1 hour of the Dyn DNS servers being down their traffic would be nearly zero. If their TTL was 86400 seconds (1 day) then a 12 hour attack would only block about half their traffic - not great, but bearable. A TTL of 1 week would mean that a 12 hour attack would be no more than an annoyance. Unfortunately, if the attack downs Dyn DNS before site owners can update their TTL this doesn't really help.

Also, the bigger a site is, the more frequently it needs to update DNS information. Twitter will serve different Internet addresses for twitter.com to users in different countries, trying to point users to the closest Twitter server to them. You don't want a user in Paris pointed to a Twitter server in San Francisco if there is one available in Amsterdam, 500 millseconds closer to them. And when you have many different servers, every day some of them are going offline for maintenance or coming online as new servers, so you need to update DNS to stop users going to the former and start sending them to the latter.

Therefore the bigger your site, the shorter your DNS TTL is likely to be, and the more vulnerable you are to this attack. If you're a small site with infrequent DNS updates, and your DNS TTL is short, then make it longer right the hell now.

Alternative designs

The alternative to this exposed address approach is to have a central service which all the baby monitors from a given manufacturer connect to, e.g. the hostname cams.babycamsRus.com; users then connect to that service as well and the service does the switching to connect Mr. and Mrs. Smith to the babycam chez Smith. This prevents the devices from being found by Internet scans - they don't have their own Internet address, and don't accept outside connections. If you can crack the BabyCams-R-Us servers then you could completely control a huge chunk of IoT devices, but their sysadmins will be specifically looking out for these attacks and it's a much more tricky proposition - it's also easy to remediate once discovered.

Why doesn't every manufacturer do this, if it's more secure? Simply, it's more expensive. You have to set up this central service, capable of servicing all your sold devices at once, and keep it running and secure for many years. In a keenly price-competitive environment, many manufacturers will say "screw this" and go for the cheaper alternative. They have no economic reason not to, no-one is (yet) prosecuting them for selling insecure devices, and customers still prefer cheap over secure.

IPv6 will make things worse

One brake on this run-away cheap-webcams-as-DoS-tool is the shortage of Internet addresses. When the Internet addressing scheme (Internet Protocol version 4, or IPv4 for short) was devised, it was defined as four numbers between 0 and 255, conventionally separated by dots e.g. This gives you just under 4.3 billion possible addresses. Back in 2006 large chunks of this address space were free. This is no longer the case - we are, in essence, out of IPv4 addresses, and there's an active trade in them from companies which are no longer using much of their allocated space. Still, getting large blocks of contiguous addresses is challenging. Even a /24 (shorthand for 256 contiguous IPv4) is expensive to obtain. Father of the Internet Vint Cerf recently apologised for the (relatively) small number of IPv4 addresses - they thought 4.3 billion addresses would be enough for the "experiment" that IPv4 was. The experiment turned into the Internet. Oops.

This shortage means that the current model where webcams and other IoT devices have their own public Internet address is unsustainable: the cost of that address will become prohibitive, and customers will need something that sits behind their single home Internet address given to them by their ISP. You can have many devices behind one address via a mechanism called Network Address Translation NAT) where the router connecting your home to the Internet lets each of your devices start connections to the Internet and allocates them a "port" which is passed to the website they connect to: when the website server responds, it sends the web page back to your router along with the port number, so the router knows which of your home devices the web page should be sent to.

The centralized service described above is (currently) the only practical solution in this case of one IP for many devices. More and more devices on the Internet will be hidden from black-hat hacker access in this way.

Unfortunately (for this problem) we are currently transitioning to use the next generation of Internet addressing - IPv6. This uses 128 bits, which is a staggering number: 340 with an additional 36 zeroes after it. Typically your ISP would give you a "/64" for your home devices to use for their public Internet addresses - a mere 18,000,000,000,000,000,000 (18 quintillion) addresses. Since there are 18 quintillion /64s in the IPv6 address space, we're unlikely to run out of them for a while even if ever person on earth is given a fresh one every day and there's no re-use.

IPv6 use is not yet mainstream, but more and more first world ISPs are giving customers IPv6 access if they want it. Give it a couple of years and I suspect high-end IoT devices will be explicitly targeted at home IPv6 setups.

Summary: we're screwed

IPv4 pressures may temporarily push IoT manufacturers to move away from publicly addressable IoT devices, but as IPv6 becomes more widely used the commercial pressures may once more become too strong to resist and the IoT devices will be publicly discoverable and crackable once more. Absent a serious improvement in secure, reliable and easy dynamic updates to these devices, the IoT botnet is here to stay for a while.


Hillary doesn't deserve to be President

I've just finished watching the #2 US Presidential Debate, chaired by Anderson Cooper - for whom I have a reasonable amount of respect as a more-fair-than-average interviewer - and Martha Raddatz, who was hopelessly out of her depth and showing awful bias. Coming out of the debate, I have one question for Hillary: how, with all the advantages you had two hours ago, did you manage to lose?

Going into this debate, Hillary had Donald cornered by the media after his not terribly edifying 2005 remarks about pussy-grabbing opportunities in showbiz were reported. Near-universal media agreement was that The Donald was fatally holed beneath the waterline. Even Trump's own Vice President pick, Pence, was publicly disapproving of Trump's comments. Republican senators and Congress critters were denouncing Trump and saying they wouldn't vote for it. In golf, this would be like being 2 inches away from the hole when your opponent is 200 yards away in a bunker, and it has just started to rain.

And yet... Hillary missed the putt, kept missing it, and Donald chipped his ball onto the green and snuck it into the hole before Hillary found her game.

Trump is not a great public speaker. His train of thought wanders as he speaks, and he assumes technical and factual knowledge in the audience rather than explaining as he goes along. These traits were in full display this evening. A great example was in the "birther" issue where Hillary accused Trump of asking the "racist" question about whether President Obama had actually been born in the USA. Trump (accurately) pointed out that this issue had first been raised by Hillary's consiglieri Sidney Blumenthal, but he did it in such an indirect way that anyone not substantially familiar with the people concerned would have had no idea what he was talking about and how it was tied to Hillary.

Still, somehow he did a better job of debating than Hillary herself. Tonight's debate format seemed to work better for him, because he's comfortable doing spontaneous exposition on topics. Hillary is awful at this, visibly working her way through pre-prepared points on each topic rather than going with the flow of the question and debate. Trump was prone to wander off the thread to include the attacks he wanted to make on Hillary (Bill's disbarring, Russia, black poverty, Syria, tax policy and of course her email server) but seemed to make most of it stick and force Hillary to respond.

Raddatz did her best to cover for Hillary's poor quality responses - Cooper, to his credit, did not - but it seemed clear to me that Trump had managed to bring up nearly all the Hillary dirty laundry that he had avoided in the first debate. Hillary did a variable job in responding to these points, but looked really weak on Russia/Syria, and her responses on the email server were strong but - frankly - flat-out lies. If Donald could learn to speak with more clarity and focus, he'd crucify her. As it was, this was a win on points only, but compared to expectations Donald killed it tonight.

Why was the pussy-grab tape such a non-event in this debate? I think it was because of the apology. Trump apologised for what he said on the tape a few hours after it was publicised, and did so again in the debate as soon as it was brought up. Once he'd done that, it was much harder for Hillary to use it as leverage. "He said these horrible things!" "I've apologised for that, you heard me." Where do you go from there? You can try "this shows what he thinks about women!" but Trump was willing to go on the offense about Bill Clinton and his bimbo eruptions - perhaps the lack of challenge in this area is a sign of how vulnerable Hillary thinks she is here.

By contrast, Hillary's mea culpa for the email server still had a whiff of "I'm sorry I got caught" - her assertions around "no evidence that anyone hacked the server" were incredibly weaselly. A responsible candidate would have agreed that it was quite likely that unfriendly nations had got at least some access to that server, and taken personal responsibility for any consequences arising from their decision to use it.

Conclusion? It's still game on for November 8th. Somehow Donald has mitigated the worst of the impact of the pussy-grab, and is challenging Hillary on the issues again. What other gotchas for him has she got left to leak? Are they good enough to be game-ending, or are they just "the same again"?


Why jail women at all?

I've noticed increasing concern among UK media column writers over the past year about the situation of women in prison, with a clamour to reduce - if not eliminate - the practice of sending women to jail. A good example is this column from Eric Allison in (where else?) the Guardian, late last year: "Women are dying in jails they should not have been sent to":

Many female prisoners are mothers and primary carers. Every year, around 18,000 children are affected by their mother being sent to jail. As women are usually the main caregiver, many end up in care. We can only guess how much that adds to the anguish of mothers behind bars.
A compelling argument to be sure.

Let us turn to the case of Eunice Spry from Gloucestershire, who was sent down for 14 years at Bristol Crown Court in 2007:

Judge Simon Darwall-Smith told the devout Jehovah's Witness that this was the worst case he had come across in 40 years.
Passing sentence, he said: "It's difficult for anyone to understand how any human being could have even contemplated what you did, let alone with the regularity and premeditation you employed."
As punishment for misbehaving, she would beat the children on the soles of their feet and force them to drink washing-up liquid and bleach.
I'm sure Eunice Spry's children were affected by her being sent to jail, but I'd imagine it's more along the lines of thanking God that she was finally kept away from them.

Her defence brief did his best to mitigate, but had something of an uphill struggle:

Mr Mitchell also revealed that Spry had needed protection in prison following her convictions and it was a "particularly unpleasant" place for her.
To which I'd be minded to respond "Et alors?" I hadn't realized before reading the detailed verdict that she was also convicted of "Intimidating a juror or witness or person assisting, or who has assisted, the investigation of an offence" - this is not just a woman who made a few bad choices.

Spry was of course eligible for parole in April 2014 and (of course) was released on schedule - the 14 year imprisonment sentence was reduced to 12 years on appeal.

There's certainly an argument that people are being sent to jail for crimes which are not obviously harmful to society - for example, possession of substantial quantities of narcotics but no obvious intent to supply outside their circle of dysfunctional friends - but let's not special-case women in this argument. If we are serious about gender equality, we should apply the same standards to the decision about jailing a father that we do about deciding to jail a mother. Otherwise we're perpetuating serious inequality in the application of the law to men and women - and isn't that something an enlightened society should want to fix?


Does Putin want Trump as President?

I'm a huge fan of thoughtful blogger Richard Fernandez from Belmont Club, but respectfully have to disagree on his take on the current Wikileaks leaking of Democratic National Catfight emails and voicemails:

By striking at Hillary's aura, the Russians may be attempting the same thing. Democratic voters looked up to her to protect and defend the nation because that's what presidents do. By hacking Hillary and humiliating her, Putin has sent the message that she cannot even defend herself -- and what's the use of a president who can't defend herself?
This is an excellent point, except that - despite the tone of publicity - Hillary is not actually President of the United States. She's locked in a deadly struggle with Donald Trump for the title, and the decision won't happen until November.

I have no trouble at all believing that the Russians have the goods on Hillary. FBI Director Comey's statement on the Clinton private email server left little doubt that any competent foreign security service would have gained complete access to her communications, and have any amount of blackmail material on her and on her confidants. But if you're playing poker and have four kings, why would you all-but-announce this at the start of bidding?

Wikileaks has doubtlessly been compromised by Russian security services, but such compromise is covert - the SVR doesn't have an editorial veto - and it still provides a low-friction platform for publicising controversial data. This is a classic example of a disgruntled insider publicising information to hurt someone they loathe; Wikileaks is just the medium.

If you doubt this assertion, ask yourself: if you were Putin, with whom would you want to negotiate? Trump who is well-established as a wildcard who could say or do anything, and is (in practice) very hard to blackmail because of all the unsavory facts which are already public? or Hillary who still tries to project an aura of robustness and foreign intelligence savvy from her time at State, and whose private email correspondence you have available on request?


I'm starting to believe that May is trolling the Guardianistas

I thought that the chorus of butthurt from the why-didn't-the-plebs-listen-to-ME part of the Remain camp was finally starting to die down, but then May appointed Johnson as Foreign Secretary, and oh my goodness. My Twitter feed and Farcebook timeline have erupted in caterwauling once again.

Note that this has the effect of focusing the limited Guardianista attention on Johnson and his various alleged[1] faux pas, and there's been very little comment on the appointment of the sharp and strongly pro-Brexit David Davis as "Minister for Brexit". I rather suspect Davis is going to be the source of most of the actual heartache for the Remainers in the next couple of years.

[1] Most of which I suspect they're overselling. Johnson has his flaws, Heaven knows, but he's a smart cookie, extremely well travelled, with a highly multinational family. And I'd endorse him as Foreign Secretary solely on the basis of his trolling of the Chinese about ping pong at the Beijing Olympics.


Denatonium Benzoate loses its crown

Also known as Bitrex, Denatonium Benzoate held the record for the most bitter substance on earth until 24th June 2016. A teaspoon of the substance added to an Olympic size swimming pool (volume 2.5M litres) makes the water noticeably bitter. Bitrex has been a very successful additive to poisonous substances to prevent accidental ingestion, such as car antifreeze.

Sadly for manufacturer Macfarlan Smith, since 24th June Bitrex's record has been overtaken by the UK Guardian opinion page. One opening paragraph has the same bitterness impact as approximately 300ml of Denatonium Benzoate. Rumours suggest that Macfarlan Smith has opened negotiations with Jonathan Freedland, Nick Cohen and Polly Toynbee for purchase of their spleens as a manufacturing source of the Bitrex successor.

It is serendipitous that the name "Bitrex" is an anagram of the new product: "Brexit".


Toys firmly out of prams

I predicted a certain amount of tantrums, but really didn't think it would get this bad this quickly. Scotland and London wanting to split off and rejoin Europe, Labour Party stalwarts gunning for Corbyn (who, up until a couple of hours ago, must have thought he'd played a blinder) and Twitter and Facebook in meltdown with Remainers calling Leavers "racist idiots" and worse.

Heavens sake, you're all adults, bloody act like it. This was a full national referendum with a turnout of 74% which is way above recent elections. If your side lost, sit down and put up with it. Don't whine like a three year old deprived of an ice cream. Leave seem to have been a heck of a lot more restrained in their unexpected win than you'd have been in their place.

Not entirely surprised by Cameron chucking the towel in. He seems to be one of the few people today (and maybe the only Remainer) acting with dignity.


Referendum predictions

I have no idea on the actual result. I don't think I could place a bet if I was offered 50:50 odds on each choice. That said, the breakdown by region is going to be very interesting, and I wonder if the rain/floods will hit turnout in the SE, and whether that will make a material difference.

If "Remain" wins: The Guardian (and, less obviously, BBC) will be insufferable. Juncker et al will keep true to their promise not to give any concessions to the UK, even if the result is knife-edge. UKIP effectively dissolves in a frenzied pit of backbiting. Who knows what the UKIP voters will do at the next election?

If "Leave" wins: Immediate witch-hunt from Guardian, BBC. Cameron resigns. Panic in Europe. Stock markets burning. Sweden and maybe Denmark start feeling popular pressure to exit or form referendum. Juncker et al refuse any trade deals with the UK. Boris's hair a fixture on the international news.

I've observed my Facebook stream becoming increasingly stridently pro-Remain over the past 2 weeks. The Leavers are keeping very quiet, presumably because they're swamped by insufferable Remainers if they post anything. Remain posts seem to be relatively free of Leaver comments. So is this due to Remain having an insurmountable majority, due to me having a supermajority of Remain friends, or because the Leavers don't care what the Remainers think or do?

Going by their selection of stories and interviewees, the BBC have steadily abandoned impartiality over the past couple of weeks. The only really studiously neutral Beebite I've seen has been the indefatigueable Kuenssberg.


Weasel will find a way

After the furore last year when it turned out that UK airport shops were demanding boarding passes to save themselves VAT but not save you any money I assumed that this was the effective end of the weasel. From my recent experience at Birmingham International (motto: "We put the 'slack jaw' in 'security'") it seems not.

First stop: the bookshop, to buy some doorstop-sized illiterate literature. No shortage of supply. I present the volume to the lady at the till who demands: "Boarding pass?" with no hint of shame. I enquire whether it's actually mandatory, at which point she rings up the transaction with no further questions. 1-0.

Next stop: W H Smith, for a magazine. Avoiding the single human-manned till I opt for the self-service till. I scan the magazine for a grand total of £2.50 - and it asks for a boarding pass, and won't proceed until I scan one. I hit the "my boarding pass won't scan" button, wait a minute for the roaming attendant to punch the override and proceed on my way. But hell, I remember the huge fuss in August 2015 about this. It seems that the airport shops were content to let the hubbub die down, then go back to their old ways.

Don't let them do this! Make them pay a cost in salaried worker time for each time they demand a boarding pass. Once the average worker salary rate times delay is more than the expected VAT, they will shut up about the boarding passes and let us buy our dubious literature un-monitored and without delay. (Until 1-2 years later when some bright MBA spark spots an opportunity to re-introduce the practice, at which point we hang them from the Heathrow radar pillar as a warning to others.)


The implications of the "Out" threats

With the UK In/Out referendum less than three weeks away, and the BetFair odds on "Leave" starting to come down - albeit still very far from 50-50 - it has been instructive to listen to the veiled, and not so veiled, threats about what will happen if the populace vote for "Leave".

A good example was the comment in late May from Jean-Claude "Piss Artist" Juncker, European Commission President:

"The United Kingdom will have to accept being regarded as a third country, which won't be handled with kid gloves.
"If the British leave Europe, people will have to face the consequences -- we will have to, just as they will. It's not a threat but our relations will no longer be what they are today."
Apparently EU officials don't want to have lengthy negotiations[1] with a Brexited UK, which makes sense. But of course, the easiest course for both sides would be to retain status quo ante: continue trade under the same conditions and tariff schedule as before. Why wouldn't this be the starting point? In general, trade tariffs hurt the populace of the country / countries that impose them: they make imported goods more expensive for their populace. The main function of trade tariffs is to protect local industry from "abuse" from "dumping" by foreign manufacturers: selling goods below the cost of local produce. This may not be good for local industry, but it's certainly good for anyone who wants to buy those goods, at least in the short term.

It seems fairly clear that, whatever the merits of the "Remain" and "Leave" positions, the EU establishment is happy to cut off its population's noses to spite the UK's faces. One has to ask: if national membership of the EU is supposed to be of benefit to the population, why would the EU take action to screw over all their population in order to punish a member nation that wanted to leave?

[1] Note that the EU can apparently spare the manpower to negotiate a mostly free trade agreeement with Canada, which has half the population and a bit more than half the GDP of the UK.