Externalities again

I just wasted half an hour of my life on the phone with my credit card company’s fraud department, as someone attempted to buy expensive tickets from an airline in Panama. Most likely my card number was compromised by Target, although it could also be due to Adobe.

It is actually surprising such breaches do not occur on a daily basis—the persons paying for the costs of a compromise (the card holder, defrauded merchants and their credit card companies via the cost of operating their fraud departments) are not the same as those paying for the security measures that would prevent the said breach, a textbook example of what economists call an externality. There are reputational costs to a business that has a major security breach, but they are occurring so often consumers are getting numbed to them.

Many states have mandatory breach disclosure laws, following California’s example. It is time for legislatures to take the next step and impose statutory damages for data breaches, e.g. $100 per compromised credit card number, $1000 per compromised social security number, and so on. In Target’s case, 40 million compromised credit cards multiplied by $100 would mean $4 billion in damages. That would make management take notice and stop paying mere lip service to security. It might also jump-start the long overdue migration to EMV chip-and-PIN cards in the United States.

The Gresham’s law of Amazon Web Services

In the bad (good?) old days when currency’s worth was established by the amount of gold or silver in coinage, kings would cut corners by debasing currency with lead, which is almost as dense as gold or silver. In the New World, counterfeiters debased gold coins with platinum, which was first smelted by pre-columbian civilizations. Needless to say, the fakes are now worth more than the originals.

The public was not fooled, however, and found ways to test coins for purity, including folkloric ones like biting a coin to see if it is made of malleable gold, rather than harder metals. People would then hoard pure gold coins, and try to rid themselves of debased coins at the earliest opportunity. This led to Gresham’s Law: bad money drives out good money in circulation.

After a year of using Amazon Web Services’ EC2 service at scale for my company (we moved to our own servers at the end of 2011), I conjecture there is a Gresham’s Law of Amazon EC2 instances – bad instances drive out good ones. Let me elaborate:

Amazon EC2 is a good way to launch a service for a startup, without incurring heavy capital expenditures when getting started and prior to securing funding. Unfortunately, EC2 is not a quality service. Instances are unreliable (we used over 80 instances at Amazon, and there was at least one instance failure a week, and sometimes up to 4). Amazon instances have poor disk I/O performance that makes them particularly unsuitable to hosting non-trivial databases (EBS is even worse, and notoriously unreliable).

Performance is also inconsistent—I routinely observed “runt” m1.large instances that performed half as well as the others. We experienced all sorts of failure modes, including disk corruptions, disks that would block forever without timing out, sporadic losses of network connectivity, and many more. Even more puzzling, I would get 50% to 70% failure rate on new instances that would not come up cleanly after being launched.

Some of this is probably due to the fact we use an uncommon OS, OpenSolaris, that is barely supported on EC2, but I suspect a big part of this is that Amazon uses low-end commodity parts, and does not proactively retire failed or flaky hardware from service. Instances that have the bad luck of being assigned to flaky hardware are more likely to fail or perform poorly, and thus more likely to be be destroyed, released and a new one reassigned in the same slot. The inevitable consequence of this is that new instances have a higher likelihood of being runts or otherwise defective than long-running ones.

One work-around is to spin up a large number of instances, test them, and destroy the poor-performing ones. AWS runts are usually correlated with slower CPU clock speeds, as older machines would be running older versions of the Xen hypervisor Amazon uses under the hood, have less cache, slower drives and so on. Iterating through virtual machines as if you are picking melons at a supermarket is a slow and painful job, however, and even their newer machines have their share of runts. We were trying to keep only machines with 2.6 or 2.66GHz processors, but more than 70% of the instances we were getting assigned were 2.2GHz runts, and it would usually take creating 5 or 6 instances on average to get a non-runt.

In the end, we migrated to our own facility in colo, because Amazon’s costs, reliability and performance were just not acceptable, and we had long passed the threshold beyond which it is cheaper to own than rent (I estimate it at $5,000 to $10,000 per month Amazon spend, depending on your workload). It is not as if other cloud providers are any better—before Amazon we had started on Joyent, which supports OpenSolaris natively, and their MTBF was in the order of 2 weeks, apparently because they replaced their original Sun hardware with substandard Dell servers and had issues with power management C-states in the Dell server BIOS.

The dirty secret of cloud services is that there is no reliable source of information on actual performance and reliability of cloud services. This brings out another economic concept, George Akerlof’s famous paper on the market for lemons. In a market where information asymmetry exists, the market will eventually collapse in the absence of guarantees. Until Amazon and others offer SLAs with teeth, you should remain skeptical about their ability to deliver on their promises.

What is heard, and what is not heard

French economist Frédéric Bastiat (1801–1850) wrote a pamphlet titled Ce qu’on voit et ce qu’on ne voit pas (What is Seen and What is Not Seen) where he demolishes the make-work fallacy in economics. When Jacques Bonhomme’s child breaks his window, paying for a replacement will circulate money in the economy, and stimulate the glassmakers’ trade. This is the visible effect. Bastiat urges us to consider what is not seen, i.e. opportunity costs, such as other, more productive uses for the money that are forgone due to the unexpected expense. This lesson is still relevant. The cost of repairing New Orleans after Katrina, or cleaning the Gulf after Deepwater Horizon, will cause a temporary boost in GDP statistics, but this is illusory and undesirable, another example of how poorly conceived metrics can distort thinking.

Another example is that of electric cars. Advocates for the blind have raised a ruckus about the dangers to blind people from quiet electric cars they cannot hear or dodge. Nissan just announced that their Leaf electric car will include a speaker and deliberately generate noise, in part to comply with the Japanese Transport Ministry’s requirements. To add injury to insult, the sound selected is apparently a sweeping sine wave, a type of sound that is incredibly grating compared to more natural sounds, including that of machinery.

Unfortunately, this is illustrates the fallacy Bastiat pointed out. Authorities are focusing on the visible (well, inaudible) first-order effect, but what is not seen matters as much. Most urban noise stems from transportation, and that noise pollution has major adverse impact on stress levels, sleep hygiene, and causes high blood pressure and cardiac problems from children to adults to the elderly. According to the WHO, for 2006 in the UK alone, an estimated 3,000+ deaths due to heart attacks can be attributed to noise pollution (out of 100,000+).

These figures are mind-boggling. For a country the size of the US, that probably comes around to five  or six 9/11 death tolls per year. Quiet electric cars should be hailed as a blessing, not a danger. There are other ways to address the legitimate concerns of the blind, e.g. by mandating transponders on cars and providing receivers for the blind.

Why do voters put up with bad politicians?

As a foreigner living in San Francisco for the last ten years, I never cease to be baffled by US voters’ tendency to vote for candidates who are clearly class warriors on the side of the rich and other influential special interests. Political scientists have long wondered why people vote against their own best interests, e.g. Americans voting for candidates beholden to health “care” provider lobbies and who hew to the status quo, saddling the US with grotesquely overpriced yet substandard health care. Another example would be the repulsive coddling of an increasingly brazen Wall Street kleptocracy.

Ideology cannot explain it all. Certainly, some people will put principle ahead of their pocketbook and vote for candidates that uphold their idea of moral values even if they simultaneously vote for economic measures that hurt their electorate. That said, there is nothing preventing a political candidate from adopting simultaneously socially conservative positions and economic policies that favor a safety net, what in Europe would be called Christian Democrats.

Media propaganda and brainwashing cannot explain it either, to believe so, as do conspiracy theorists on both right and left of the US political spectrum, is to seriously underestimate the intelligence (and cynicism) of the electorate. In a mostly democratic country like the United States, special interests can only prevail when the general population is apathetic, or at least consents to the status quo.

I believe the answer lies in loss aversion, the mental bias that causes people to fear a loss far more than they desire a gain. Our brains did not evolve in a way that favors strict rationality. Most people’s intuition about probability and statistics is unreliable and misleading—we tend to overestimate the frequency of rare events. The middle class, which holds a majority of votes, will tend to oppose measures that expose it to the risk of being pulled down by lower classes even if the same measures would allow them upward mobility into the upper classes. The upper class exploits this asymmetry to maintain its privileges, be they obscene taxpayer-funded bonuses for bankers who bankrupted their banks, or oligopoly rent-seeking by the medical profession.

Why I will never buy a Kindle

One of my bosses got a Kindle 2 a few months ago, and was wondering how an avowed gadget lover such as myself did not have one already. I am perfectly comfortable reading books in electronic form on the small screens of PDAs or phones, but I have little interest in carrying yet another device with its bevy of chargers and accessories, so I just humored him. As far as I am concerned, the Apple iPad pretty much killed the e-reader market. E-ink technology has a place in digital signage, but a general-purpose computing device with Internet connectivity like the iPad wins over a unitasker any day.

My main objection to commercial e-books as they are mooted today is digital rights management. e-books cannot be resold or even given to family members. Even if DRM were acceptable, the value of a restricted e-books is a fraction of the value of a real book, but pricing today is much higher, despite massively lower costs of production, and short-sighted publishers want to take them even higher, to the same levels as hardbacks.

All tech companies fall somewhere on a spectrum of evil. Microsoft is on the bumbling side—their products are inferior and their marketing practices sharp, to say the least, but they are a fairly open company when it comes to developers using their platform, and Bill Gates is a modern day Robin Hood of sorts, taking from rich Westerners and giving to the poor in the Third World. Apple embodies the seductive dark side—superior products but a company that has no compuction in stabbing developers in the back, and with a demonstrated penchant for control freakery as shown with the iPhone App Store. Google is on the undecided side, ruthlessly violating privacy, but still capable of the odd principled gesture such as facing down Chinese censors.

Amazon as a company lies quite far on this spectrum. Good customer service does not excuse their behavior:

  • Jeff Bezos is personally listed as an inventor on the obviously frivolous “one click” patent and has been using it to extort royalties and stymie competitors.
  • At one point they removed all gay themed books from their search listings by classifying them. Faced with a firestorm of controversy, they unconvincingly claimed it was an operator error. Why do they have a bulk blacklisting facility in the first place?
  • In an example of life imitating art, they pulled e-book copies of Orwell’s “Nineteen Eighty Four” from Kindle users who had paid for them. Apparently, they had never bothered to check if they had the rights to sell them. The simple fact Amazon has the power to pull books back from electronic bookshelves is unacceptable.
  • They are trying to leverage their dominant position in online book sales to monopolize print-on-demand publishing by refusing to carry books not published by their own on-demand imprint, BookSurge, even though the latter is higher priced than competition and has serious quality issues.
  • This is only the tip of the iceberg. Publishers speak in hushed tones about Amazon’s thuggish “negotiating” tactics, but never publicly out of fear of retaliation.

Since the launch of the Kindle, which is estimated to have 70% market share in e-readers, Amazon has been trying to leverage its market power in paper book sales to corner the market in e-books. One of the prongs in their strategy is to keep the legacy model where the publisher treats the e-book store like a dead-tree book reseller, rather than a model and revenue share more in line with the true costs of e-books (which are obviously much lower than for physical books, as the bandwidth required is piddling).

Apple’s iPad and its associated iBooks store has changed the way the debate is framed, and offers publishers an attractive agency model to counter Amazon’s diktat. It is not surprising that five of the big six publishers (all but Random House) signed up for the iBooks store.

Last Friday, in an escalation of mind-boggling arrogance, Amazon decided to punish Macmillan, the smallest and weakest of the big six (at least in the US) by withdrawing every Macmillan book from sale, including paper books, not just e-books. Among others books by Macmillan affiliate Tor, the leading label in Science Fiction and Fantasy, are not available for sale by Amazon (although they are still available from third-party sellers via Amazon’s site). Essentially Amazon is trying to use its dominance in printed book sales to twist Macmillan’s arm. As far as I am concerned, this is racketeering.

Disclaimer: my wife used to work for Macmillan in the UK. Not that it matters, Amazon’s behavior would be just as reprehensible with any other publisher.

I do not approve of the publishing industry’s doomed attempts to impose premium pricing on e-books, or their attempts to impose unacceptable DRM, but customers are perfectly capable of voting with their feet, as I do, and a middleman like Amazon behaving this way is intolerable. Booksellers censoring books or limiting supply is not an innocuous act. Norman Spinrad is in self-imposed exile in Paris because B. Dalton and Waldenbooks, the dominant booksellers in the 80s, would not sell his more controversial books (like Journals of the Plague Years) out of fear of offending conservative audiences in the Bible Belt.

Small independent bookstores are failing everywhere, and even the large Barnes & Noble and Borders chains are in dire straits. A company like Amazon with a demonstrated history of abusing its market power cannot be permitted to continue. I always buy my SFF books from the lovely Borderlands Books in any case, and my classical CDs from Arkiv Music, but I will henceforth abstain from buying books from Amazon altogether.

As for the Kindle, it can go to hell. I would not take one if they gave it to me for free.

Update (2010-02-04):

Like the SFWA, I replaced all the Amazon links on this site to Indiebound, a website that helps support independent booksellers.

Update (2014-05-28):

They are employing their racketeering tactics again, this time against Hachette.

City government waste in San Francisco

A 33% hike in Muni fares was announced today. This will hit the poorest people in the city first, and to add insult to injury, this is accompanied with cuts in service.

San Francisco has a budget of over $6B, about the same size as much larger cities as Chicago or Paris, and exceeding the budget of 20 of the US states. It also exceeds the entire GNP of countries like Mongolia or Georgia (in the Caucasus). San Franciscans get little to show for it in services.

One reason why: SF has over 8,000 city employees making over $100,000 a year (the head of Muni is one of them, making $325,000, or more than US Cabinet ministers who make $191,000). The share of the city budget spent on those high flyers is over $1B…

Real-world organizational change

The economic crisis headlines are rife with CEOs and other executives behaving in ways contrary to the interests of shareholders. Business schools teach agency theory, an economic discipline that aims to solve the problem where an agent of principals (e.g. shareholders) needs to be given wide power to perform his job, but could abuse these powers in ways inimical to the principal’s interests, specially when accountability is difficult because the agent controls the information fed back to the principals. The standard recommendation of agency theory is to align the agent’s incentives with those of the principal by setting performance-based compensation such as stock option schemes.

In the real world, of course, the reaction of an agent given a set of incentives is not to act in the way they were designed (too much work) but rather to game the system. In the case of stock options, take large amounts of risk that mostly pays off, but in unlikely cases fails catastrophically, the opposite of insurance, in fact. In the most likely case, they deliver apparently excellent results, and pocket handsome bonuses. When the bets turn spectacularly bad, the shareholders foot the bill. The failure of agency theory is just another example of how intellectually bankrupt and disconnected from reality most B-School curricula are.

This is not a new problem. Historical rulers from Cyrus the Great to Alexander to Peter the Great to Stalin had to control power-hungry and untrustworthy satraps or barons. The way they solved the problem was invariably to set up a network of spies and run two parallel and competing chains of command constantly at each other’s throats, to keep each other honest. To a certain extent, the more arbitrary and capricious the ruler’s favor seems to be, the more effective it is at keeping the underlings in line.

Investors would be well advised to study the tried and proven methods of the great tyrants. The way to control untrustworthy agents is fear and setting up parallel information-gathering networks unbeholden to the agent, not by lavishing rewards that only motivate even worse behavior. Instead of relying on corporate governance consultancies like ISS, they should hire private eyes like Pinkerton to infiltrate their companies.

The Tropicana redesign: marketing genius?

One of the great things about living in the United States is the ubiquity and affordability of high-quality orange juice. A few weeks ago, while going through the aisles at Target to buy a carton of Tropicana, I couldn’t find any. It looked like Target had replaced them with cheap generic knock-offs. On closer inspection, it turned out Pepsi redesigned the packaging. To say the new design is ugly is an understatement. Many comments on the redesign compare it to generics in its amateurism.

Pepsi’s marketers are not legends in the field like those of Procter & Gamble, but still, I find it hard to believe no one there perceived how bad the new cartons look, and how off-putting they are. This led me to think if this wasn’t intentional.

In this economy, sales of Spam are exploding even though the ersatz canned meat is actually more expensive than more nutritive fresh meat and a much worse value. One explanation is that spam is what economists call an inferior good, a good for which demand increases as incomes decrease because people can’t afford the better stuff. One extreme type of an inferior good is a Giffen good, a product for which demand increases even as its price increases. Economists still debate whether Giffen goods even exist. One often-quoted (and just as often disputed) example are potatoes during the Great Irish famine of 1847. As the price of potatoes rose, the poor were locked in a vicious circle of not being to afford anything else and being more dependent on potatoes, which only accelerated the price explosion.

Perhaps the Tropicana marketers have figured that in a severely down economy, people are settling for inferior goods, and making Tropicana look downmarket may increase its sales…

Update (2009-02-23):

So much for my theory. Although you could make the case this is more like the New Coke fiasco (that many conspiracy theorists still think was deliberate).

The tail wagging the dog redux

I don’t understand why the media make such a big fuss about computer companies like Apple, Dell, Gateway or now Cisco entering the consumer electronics business. Consumer electronics is a puny industry compared to IT. Just compare the market cap and revenues of Apple, HP or Cisco to that of Sony.

Diversifying into consumer electronics is a no-brainer because entertainment’s migration to digital makes many technologies dual-purpose, but consumer electronics will remain a marginal sideline to IT companies’ business, not a replacement growth engine.

A similar instance of the media industry’s myopia (and indeed narcissism) is how they make a big fuss of telecom and cable companies entering the market for music – the entire content industry’s revenues, music movies et al, does not amount to more than a couple weeks’ revenues for telcos.

Vignettes from India

I have been in Bombay for a week now. Traffic snarl-ups are a reality of life here, and enterprising hawkers flog magazines to passengers caught in traffic jams.

According to my wife, one of the publications they sell is “ee-keya” – Ikea catalogs. The thing is, Ikea does not (yet) have a presence in India. What people do is buy the catalog, then ask their carpenters to reproduce the furniture therein. It seems Ikea design is quite prestigious here…

The benefits of standardization

In France, nuclear power plants follow a standardized design. The end result is that France gets more than 70% of its electricity from nuclear, reducing its dependence on oil or natural gas from the unsavory regimes in Russia or Algeria, and making it a leader in reducing greenhouse gas emissions. The lessons from security incidents are also immediately applied to all other plants, leading to higher overall safety. Technicians trained on one plant are qualified for all, leading to more flexibility in managing the workforce.

In the US, each nuclear plant is a one-off, with the results being much higher costs to construct and operate them.

The same situation applies to the by-laws of public companies. Each is hand-crafted. The only beneficiaries of this situation are lawyers and crooked executives who use their mastery of the arcana in by-laws to prevent shareholders from exercising their rights.

Public company rules should be defined by statute, not left to the discretion of executives and their tame boards.


The press recycles worn clichés when it reports on volatility in the markets. Last week’s 700-point drop was a “collapse” and today’s 5% drop is a “meltdown”.

I’m sorry, but 30% is a meltdown and 50% is a collapse. 5% or 7% do not rate such excessive terms.

Forensic accounting automation

A widespread financial meltdown almost invariably correlates with widespread fraud. Companies that specialize in data mining should focus on developing forensic accounting data mining tools instead, and partner with investigation agencies or law firms to sell audit and prosecution services to the government, on a bounty basis. It’s not as if the enterprise market is going anywhere but down.

Financial integrals

Much of the current financial crisis is due not so much to realized losses on mortgage-backed securities as on uncertainty as to how much the derivatives are worth. Perhaps it’s time to start reversing the process and producing individual mortgages back from derivatives, which at least have some assessable value, even if real estate prices fall by 30%.

The Albanian scenario

People are only now beginning to realize the real estate bubble of the noughties was naught but a gigantic pyramid scheme. There is unexpected resistance to the idea of bailing out the investment bankers who did most to get us in this mess (while paying themselves handsomely to do so), and one of the counter-proposals is to give money to insolvent mortgage owners, i.e. reward the imprudent over those who followed the rules, did not lie about their income on a loan application. Economists call this moral hazard.

That said, the idea may have political wings. Investment bankers are not the only ones who like the idea of feeding at the public through. When a substantial enough proportion of the population loses its shirt in a pyramid scheme, it expects to be compensated by the public purse, and sometimes the entire social order breaks down, as happened in Albania circa 1990. Might this be the direction the US is headed towards?

The CEO should not be a member of the board

It’s an open secret that most public corporations exhibit abysmal corporate governance. Insider management is skilled in using corporate by-laws against the shareholders that are nominally their bosses. One good example is HP’s acquisition of Compaq. Whatever the merits of the deal, the company (i.e. Carly Fiorina) spent half a billion dollars of shareholder’s money in PR expenditures opposed by a significant chunk of the said shareholders, lobbying for an outcome that would handsomely profit the same executives with retention bonuses, a flagrant case of self-dealing.

Most shares are owned by large institutional investors that are too lazy to do proper due diligence. In many case, they are pension funds or investment banks that curry management’ favor in the form of contracts. Index funds can’t even vote with their feet. The only way out would be for corporate by-laws to be standardized and made statutory, rather than one-offs rife with potential for abuse.

A simple question: if a board’s job is to hire and fire a company’s CEO, why is the CEO, who is after all a mere employee, allowed to participate in the board at all, let alone preside it? The CEO should be accountable to the board, not a member of it, or privy to the board’s deliberations. This is in part due to gross over-hyping of CEOs’ importance. What they do is neither unique or as rare a skill as is often supposed, and their prima donna demands should be resisted. Of course, most companies’ boards today are already dominated by their management, so the rot is set too deep to expunge easily.

I concluded years ago that most public corporations are run by self-dealing kleptocracies. They loot most of the companies’ profits, and leave some crumbs to the shareholders. The only way to realize the true potential of investing in a profit-making enterprise is to be an active majority shareholder, either by buying a controlling stake (an option available only to those already wealthy) or by starting one yourself. Belgian financier Albert Frère had a saying “The difference between a big minority shareholder and a small minority shareholder is the difference between a big chump and a small chump”.

Amazon wishlist optimizer

I wrote a script several months ago to go through an Amazon wish list and find the combination of items that will best fit within a given budget. Given that the Christmas holiday shopping season seems to have started before Thanksgiving, it seemed topical to release it.

It used the Amazon Web Services API, which is a complete crock (among other failings, it will consistently not return the price for an item, even when explicitly instructed to do so). It does not look like Amazon pays any particular attention to the bug reports I filed. I just gave up on the API and re-implemented it the old-fashioned way, by “scraping” Amazon’s regular (and most definitely not XML-compliant) HTML pages.

It is still very much work in progress, but already somewhat useful. You can use it directly by stuffing your wish list ID in the URL (or using the form below):

Wish list IDAmount

A better way is to drag and drop the highlighted Amazon optimizer bookmarklet link (version 6 as of 2007-05-08) to your browser’s toolbar. You can then browse through Amazon, and once you have found the wish list you are looking for, click on the bookmarklet to open the optimizer in a new window (or tab). By default, it will try and fit a budget of $100 (my decadent tastes are showing, are they not?), but you can change that amount and experiment with different budgets. Surprisingly often, it will find an exact fit. Otherwise, it will try to find the closest match under the budget with as little left over as possible.

There are many caveats. The wishlist optimizer only works for public (US) wish lists. There does not seem to be an easy way to buy multiple items for somebody else’s wish list in one step, although I am working on it, so you will have to go through the wish list and add the items by hand. Shipping costs and wish list priorities are currently not taken into account. Sometimes Amazon will not show a price straight away but instead require you to click on a link, the optimizer will decline to play these marketer’s games and just skip those products.

Be patient – is rather slow right now — it seems they did not learn the lessons of their poor performance towards the end of last year. One of my coworkers ran the optimizer through an acid test with his wife’s 13-page wish list, and it took well over a minute and half to fetch the list, let alone optimize it. One can only imagine how bad it will get when the Christmas shopping season begins in earnest. To mitigate this somewhat, I have added caching – the script will only hit Amazon once per hour for any given wish list. As it works by scraping the web site rather than using the buggy and unreliable Amazon Web Services API, there is a real risk it will stop working if Amazon blocks my server’s IP or if they radically change their wish list UI (they would do better to add additional machines and load-balancers, but that would be too logical).

Update (2005-12-02):

Predictably, Amazon changed their form (they changed the form name from edit-items to editItems) and broke not only the wishlist optimizer, but also the bookmarklet. I fixed this and upgraded to the scraping module BeautifulSoup, but you will need to use the revised bookmarklet above to make it work again.

Update (2010-04-27):

The script has been broken for quite a while, but I fixed it and it should work again.