Externalities again

I just wasted half an hour of my life on the phone with my credit card company’s fraud department, as someone attempted to buy expensive tickets from an airline in Panama. Most likely my card number was compromised by Target, although it could also be due to Adobe.

It is actually surprising such breaches do not occur on a daily basis—the persons paying for the costs of a compromise (the card holder, defrauded merchants and their credit card companies via the cost of operating their fraud departments) are not the same as those paying for the security measures that would prevent the said breach, a textbook example of what economists call an externality. There are reputational costs to a business that has a major security breach, but they are occurring so often consumers are getting numbed to them.

Many states have mandatory breach disclosure laws, following California’s example. It is time for legislatures to take the next step and impose statutory damages for data breaches, e.g. $100 per compromised credit card number, $1000 per compromised social security number, and so on. In Target’s case, 40 million compromised credit cards multiplied by $100 would mean $4 billion in damages. That would make management take notice and stop paying mere lip service to security. It might also jump-start the long overdue migration to EMV chip-and-PIN cards in the United States.

The Gresham’s law of Amazon Web Services

In the bad (good?) old days when currency’s worth was established by the amount of gold or silver in coinage, kings would cut corners by debasing currency with lead, which is almost as dense as gold or silver. In the New World, counterfeiters debased gold coins with platinum, which was first smelted by pre-columbian civilizations. Needless to say, the fakes are now worth more than the originals.

The public was not fooled, however, and found ways to test coins for purity, including folkloric ones like biting a coin to see if it is made of malleable gold, rather than harder metals. People would then hoard pure gold coins, and try to rid themselves of debased coins at the earliest opportunity. This led to Gresham’s Law: bad money drives out good money in circulation.

After a year of using Amazon Web Services’ EC2 service at scale for my company (we moved to our own servers at the end of 2011), I conjecture there is a Gresham’s Law of Amazon EC2 instances – bad instances drive out good ones. Let me elaborate:

Amazon EC2 is a good way to launch a service for a startup, without incurring heavy capital expenditures when getting started and prior to securing funding. Unfortunately, EC2 is not a quality service. Instances are unreliable (we used over 80 instances at Amazon, and there was at least one instance failure a week, and sometimes up to 4). Amazon instances have poor disk I/O performance that makes them particularly unsuitable to hosting non-trivial databases (EBS is even worse, and notoriously unreliable).

Performance is also inconsistent—I routinely observed “runt” m1.large instances that performed half as well as the others. We experienced all sorts of failure modes, including disk corruptions, disks that would block forever without timing out, sporadic losses of network connectivity, and many more. Even more puzzling, I would get 50% to 70% failure rate on new instances that would not come up cleanly after being launched.

Some of this is probably due to the fact we use an uncommon OS, OpenSolaris, that is barely supported on EC2, but I suspect a big part of this is that Amazon uses low-end commodity parts, and does not proactively retire failed or flaky hardware from service. Instances that have the bad luck of being assigned to flaky hardware are more likely to fail or perform poorly, and thus more likely to be be destroyed, released and a new one reassigned in the same slot. The inevitable consequence of this is that new instances have a higher likelihood of being runts or otherwise defective than long-running ones.

One work-around is to spin up a large number of instances, test them, and destroy the poor-performing ones. AWS runts are usually correlated with slower CPU clock speeds, as older machines would be running older versions of the Xen hypervisor Amazon uses under the hood, have less cache, slower drives and so on. Iterating through virtual machines as if you are picking melons at a supermarket is a slow and painful job, however, and even their newer machines have their share of runts. We were trying to keep only machines with 2.6 or 2.66GHz processors, but more than 70% of the instances we were getting assigned were 2.2GHz runts, and it would usually take creating 5 or 6 instances on average to get a non-runt.

In the end, we migrated to our own facility in colo, because Amazon’s costs, reliability and performance were just not acceptable, and we had long passed the threshold beyond which it is cheaper to own than rent (I estimate it at $5,000 to $10,000 per month Amazon spend, depending on your workload). It is not as if other cloud providers are any better—before Amazon we had started on Joyent, which supports OpenSolaris natively, and their MTBF was in the order of 2 weeks, apparently because they replaced their original Sun hardware with substandard Dell servers and had issues with power management C-states in the Dell server BIOS.

The dirty secret of cloud services is that there is no reliable source of information on actual performance and reliability of cloud services. This brings out another economic concept, George Akerlof’s famous paper on the market for lemons. In a market where information asymmetry exists, the market will eventually collapse in the absence of guarantees. Until Amazon and others offer SLAs with teeth, you should remain skeptical about their ability to deliver on their promises.

What is heard, and what is not heard

French economist Frédéric Bastiat (1801–1850) wrote a pamphlet titled Ce qu’on voit et ce qu’on ne voit pas (What is Seen and What is Not Seen) where he demolishes the make-work fallacy in economics. When Jacques Bonhomme’s child breaks his window, paying for a replacement will circulate money in the economy, and stimulate the glassmakers’ trade. This is the visible effect. Bastiat urges us to consider what is not seen, i.e. opportunity costs, such as other, more productive uses for the money that are forgone due to the unexpected expense. This lesson is still relevant. The cost of repairing New Orleans after Katrina, or cleaning the Gulf after Deepwater Horizon, will cause a temporary boost in GDP statistics, but this is illusory and undesirable, another example of how poorly conceived metrics can distort thinking.

Another example is that of electric cars. Advocates for the blind have raised a ruckus about the dangers to blind people from quiet electric cars they cannot hear or dodge. Nissan just announced that their Leaf electric car will include a speaker and deliberately generate noise, in part to comply with the Japanese Transport Ministry’s requirements. To add injury to insult, the sound selected is apparently a sweeping sine wave, a type of sound that is incredibly grating compared to more natural sounds, including that of machinery.

Unfortunately, this is illustrates the fallacy Bastiat pointed out. Authorities are focusing on the visible (well, inaudible) first-order effect, but what is not seen matters as much. Most urban noise stems from transportation, and that noise pollution has major adverse impact on stress levels, sleep hygiene, and causes high blood pressure and cardiac problems from children to adults to the elderly. According to the WHO, for 2006 in the UK alone, an estimated 3,000+ deaths due to heart attacks can be attributed to noise pollution (out of 100,000+).

These figures are mind-boggling. For a country the size of the US, that probably comes around to five  or six 9/11 death tolls per year. Quiet electric cars should be hailed as a blessing, not a danger. There are other ways to address the legitimate concerns of the blind, e.g. by mandating transponders on cars and providing receivers for the blind.

Why do voters put up with bad politicians?

As a foreigner living in San Francisco for the last ten years, I never cease to be baffled by US voters’ tendency to vote for candidates who are clearly class warriors on the side of the rich and other influential special interests. Political scientists have long wondered why people vote against their own best interests, e.g. Americans voting for candidates beholden to health “care” provider lobbies and who hew to the status quo, saddling the US with grotesquely overpriced yet substandard health care. Another example would be the repulsive coddling of an increasingly brazen Wall Street kleptocracy.

Ideology cannot explain it all. Certainly, some people will put principle ahead of their pocketbook and vote for candidates that uphold their idea of moral values even if they simultaneously vote for economic measures that hurt their electorate. That said, there is nothing preventing a political candidate from adopting simultaneously socially conservative positions and economic policies that favor a safety net, what in Europe would be called Christian Democrats.

Media propaganda and brainwashing cannot explain it either, to believe so, as do conspiracy theorists on both right and left of the US political spectrum, is to seriously underestimate the intelligence (and cynicism) of the electorate. In a mostly democratic country like the United States, special interests can only prevail when the general population is apathetic, or at least consents to the status quo.

I believe the answer lies in loss aversion, the mental bias that causes people to fear a loss far more than they desire a gain. Our brains did not evolve in a way that favors strict rationality. Most people’s intuition about probability and statistics is unreliable and misleading—we tend to overestimate the frequency of rare events. The middle class, which holds a majority of votes, will tend to oppose measures that expose it to the risk of being pulled down by lower classes even if the same measures would allow them upward mobility into the upper classes. The upper class exploits this asymmetry to maintain its privileges, be they obscene taxpayer-funded bonuses for bankers who bankrupted their banks, or oligopoly rent-seeking by the medical profession.

Why I will never buy a Kindle

One of my bosses got a Kindle 2 a few months ago, and was wondering how an avowed gadget lover such as myself did not have one already. I am perfectly comfortable reading books in electronic form on the small screens of PDAs or phones, but I have little interest in carrying yet another device with its bevy of chargers and accessories, so I just humored him. As far as I am concerned, the Apple iPad pretty much killed the e-reader market. E-ink technology has a place in digital signage, but a general-purpose computing device with Internet connectivity like the iPad wins over a unitasker any day.

My main objection to commercial e-books as they are mooted today is digital rights management. e-books cannot be resold or even given to family members. Even if DRM were acceptable, the value of a restricted e-books is a fraction of the value of a real book, but pricing today is much higher, despite massively lower costs of production, and short-sighted publishers want to take them even higher, to the same levels as hardbacks.

All tech companies fall somewhere on a spectrum of evil. Microsoft is on the bumbling side—their products are inferior and their marketing practices sharp, to say the least, but they are a fairly open company when it comes to developers using their platform, and Bill Gates is a modern day Robin Hood of sorts, taking from rich Westerners and giving to the poor in the Third World. Apple embodies the seductive dark side—superior products but a company that has no compuction in stabbing developers in the back, and with a demonstrated penchant for control freakery as shown with the iPhone App Store. Google is on the undecided side, ruthlessly violating privacy, but still capable of the odd principled gesture such as facing down Chinese censors.

Amazon as a company lies quite far on this spectrum. Good customer service does not excuse their behavior:

  • Jeff Bezos is personally listed as an inventor on the obviously frivolous “one click” patent and has been using it to extort royalties and stymie competitors.
  • At one point they removed all gay themed books from their search listings by classifying them. Faced with a firestorm of controversy, they unconvincingly claimed it was an operator error. Why do they have a bulk blacklisting facility in the first place?
  • In an example of life imitating art, they pulled e-book copies of Orwell’s “Nineteen Eighty Four” from Kindle users who had paid for them. Apparently, they had never bothered to check if they had the rights to sell them. The simple fact Amazon has the power to pull books back from electronic bookshelves is unacceptable.
  • They are trying to leverage their dominant position in online book sales to monopolize print-on-demand publishing by refusing to carry books not published by their own on-demand imprint, BookSurge, even though the latter is higher priced than competition and has serious quality issues.
  • This is only the tip of the iceberg. Publishers speak in hushed tones about Amazon’s thuggish “negotiating” tactics, but never publicly out of fear of retaliation.

Since the launch of the Kindle, which is estimated to have 70% market share in e-readers, Amazon has been trying to leverage its market power in paper book sales to corner the market in e-books. One of the prongs in their strategy is to keep the legacy model where the publisher treats the e-book store like a dead-tree book reseller, rather than a model and revenue share more in line with the true costs of e-books (which are obviously much lower than for physical books, as the bandwidth required is piddling).

Apple’s iPad and its associated iBooks store has changed the way the debate is framed, and offers publishers an attractive agency model to counter Amazon’s diktat. It is not surprising that five of the big six publishers (all but Random House) signed up for the iBooks store.

Last Friday, in an escalation of mind-boggling arrogance, Amazon decided to punish Macmillan, the smallest and weakest of the big six (at least in the US) by withdrawing every Macmillan book from sale, including paper books, not just e-books. Among others books by Macmillan affiliate Tor, the leading label in Science Fiction and Fantasy, are not available for sale by Amazon (although they are still available from third-party sellers via Amazon’s site). Essentially Amazon is trying to use its dominance in printed book sales to twist Macmillan’s arm. As far as I am concerned, this is racketeering.

Disclaimer: my wife used to work for Macmillan in the UK. Not that it matters, Amazon’s behavior would be just as reprehensible with any other publisher.

I do not approve of the publishing industry’s doomed attempts to impose premium pricing on e-books, or their attempts to impose unacceptable DRM, but customers are perfectly capable of voting with their feet, as I do, and a middleman like Amazon behaving this way is intolerable. Booksellers censoring books or limiting supply is not an innocuous act. Norman Spinrad is in self-imposed exile in Paris because B. Dalton and Waldenbooks, the dominant booksellers in the 80s, would not sell his more controversial books (like Journals of the Plague Years) out of fear of offending conservative audiences in the Bible Belt.

Small independent bookstores are failing everywhere, and even the large Barnes & Noble and Borders chains are in dire straits. A company like Amazon with a demonstrated history of abusing its market power cannot be permitted to continue. I always buy my SFF books from the lovely Borderlands Books in any case, and my classical CDs from Arkiv Music, but I will henceforth abstain from buying books from Amazon altogether.

As for the Kindle, it can go to hell. I would not take one if they gave it to me for free.

Update (2010-02-04):

Like the SFWA, I replaced all the Amazon links on this site to Indiebound, a website that helps support independent booksellers.

Update (2014-05-28):

They are employing their racketeering tactics again, this time against Hachette.