On the bugginess of El Capitan

I never updated my home Mac Pro to El Capitan. To paraphrase Borges, each successive Apple OS release since Snow Leopard makes you long for the previous one. Unfortunately I have no choice but to run the latest OS X release on my work Macs as that is usually required to run the latest Xcode, itself required for the App Store.

I did not realize how bad El Capitan was until I upgraded my work iMac (27-inch 5K model) to Sierra last week. Previously, I would experience a mean time between crashes of around 3 days. I thought it was flaky hardware (the problems started from when I unboxed the computer), but couldn’t find the time to take it to the Genius Bar. I had also experienced the same problem with my old home 2009 Nehalem Mac Pro, which I had taken to the office, in fact that’s why I bought the iMac in the first place (and the first one I ordered had to go back because of defective pins in the RAM expansion slots). The Mac Pro had previously been rock-steady at home.

Since I upgraded to Sierra, I haven’t had a single crash. The only possible conclusion is that El Capitan bugs were to blame. The only thing unusual about this iMac is I upgraded the RAM from OWC, but the memory passes testing using Micromat’s TechTool.

I am not one to look at the Steve Jobs era with rosy-tinted glasses, OS X has never had the same level of stability as Solaris or even Linux, but Apple’s hardware and software quality has really gone to the dogs of late, something Lloyd Chambers dubs Apple Core Rot.

I am now starting to hedge my bets and am testing Ubuntu for my laptop computing needs, first by repurposing my 2008-vintage first-generation MacBook Air that is no longer supported by OS X anyway (works, but painfully slow) and soon with a shiny new HP Spectre on order.

Avoiding counterfeit goods on Amazon: mission impossible?

I mentioned previously that I seldom shop for electronics on any more, preferring B&H Photo whenever possible. I now have another reason: avoiding counterfeit goods.

My company boardroom is in an electromagnetic war zone—dozens of competing WiFi access points combined with electronic interference from the US-101 highway just outside make WiFi reception tenuous at best, and unusable more often than not. To work around this, we set up a wired Ethernet switch, and since most of our staff use MacBook Airs, Apple USB Ethernet adapters purchased from Amazon. When I side-graded from my 15″ Retina MacBook Pro to a much more portable 12″ Retina MacBook, I wasn’t able to connect using the dongle, and the name of the device was interspersed with Chinese characters. At first I thought it was an issue with my Satechi USB-C hub, but I experienced the same problems via a genuine Apple USB-C multiport adapter as well.

Eventually I figured out the Ethernet dongles were counterfeit. The packaging, while very similar to Apple’s, was just a tiny bit off, like amateurish margins between the Apple logo and the edges of the card. On the dongles themselves, the side regulatory disclosures sticker was inset, not flush with the body of the adapter.

Counterfeiting is a major problem. By some accounts, one third of all Sandisk memory cards worldwide are counterfeits. In some cases like chargers or batteries, your equipment could be at risk, or even your very life. The counterfeit adapters we purchased from Amazon did not come from Amazon themselves but from a third-party merchant participating in the Amazon marketplace. To Amazon’s credit, we returned them for a prompt, no questions asked refund even though we bought them over six months ago, but it is hard to believe Amazon is unaware of the problem rather than willfully turning a blind eye to it.

My first reaction was to tell our Office Manager to make sure to buy only from Amazon rather than third-party merchants (pro tip: including “amazon” in your Amazon search terms will do that in most cases). Unfortunately, that may not be enough. Amazon has a “fulfilled by Amazon” program for merchants where you ship your goods to them, and they handle warehousing and fulfillment. These “fulfilled by Amazon” items are also more attractive to Prime members. One option Amazon offers is Stickerless, commingled inventory where the items you send are put into a common bin. Amazon still has the ability to trace the provenance of the item through its inventory management, but for purposes of order fulfillment they will be handled just like Amazon’s own stock. Some categories like groceries and beauty products are excluded, but electronics are not.

The implications are huge: even if the vendor is Amazon itself, you cannot be sure that the item is not counterfeit. All the more reason to buy only from trustworthy, single-vendor sites like B&H, even if shipping is a bit slower.

ArtisanState review

TL:DR—avoid them.

I seldom print photos any more. When I do, I prefer to make photobooks, as the format is way more convenient than loose prints, takes little space, and looks more polished than a traditional photo album.

Unfortunately, most photobooks are printed on HP Indigo digital presses, which use a technology somewhat similar to a laser printer, but capable of better quality photo reproduction. Indigo presses were originally designed to produce personalized junk-mail, not high-quality photo reproduction, and the quality, while decent, is not at the same level as that of true RA-4 photo paper exposed with a laser or LED light source as done by most digital minilabs (e.g. Fuji Frontier or Noritsu QSS) or higher-end imaging systems like the Océ/Cymbolic Lightjet or Durst Lambda.

There are higher-quality options. AdoramaPix has a good reputation for its albums, which are printed on RA-4 paper and bound in a lay-flat binding without a gutter, a technique that lends itself specially well to panoramic prints. They also have a “Hudson” line of premium albums where the photos are laminated on thick cardstock for a more luxurious feel. In researching this flush-mount process, I discovered a company called ArtisanState. It is based here in San Francisco (manufacturing is in China), their pricing seemed attractive, so I decided to give them a try.

I used a selection of my panoramic prints and ordered a 6×8 album bound in genuine leather. They offer two grades of paper, Fuji Crystal Archive Lustre and Fuji Crystal Archive Pearl Metallic, and I opted for the latter. Metallic paper, first introduced by Kodak under the Endura brand, has mica particles embedded in the RC paper base under the photographic emulsion. The photo looks as if it is painted on metal, which can be spectacular, specially with specular highlights (although I would not recommend it for portraiture such as a wedding album, the fashion industry seems to be quite fond of it). The Fuji lustre has a grainy finish that resists fingerprints, but I don’t find it attractive, and would much prefer a satin finish without an obvious texture like the one Moo uses in its business cards.

When I received my album after 2 weeks, I was impressed by the reproduction quality and the metallic effect, but there was also a very visible texture on the pages, similar to an orange peel. After some research, I found that mounting metallic papers seems to cause orange peel unless done very carefully using a low temperature on the mounting press, and they are the exception to the general rule of thumb that Fuji products are superior to Kodak’s (although true to form, Kodak’s bean counters degraded the quality of the product by cutting corners to shave costs).

At the price they charge ($104 list, but I got it at 40% promotional discount), you can rightfully expect perfection. I wrote to ArtisanState to complain, got the run-around, and reviews online suggest my experience with unresponsive support is far from unusual. I am going to try again with AdoramaPix: they may be more expensive, but the product won’t be made in China and in the end you get what you pay for.

The slow decline of Amazon Prime

I have been an Amazon Prime customer since it was introduced, almost a decade ago in 2005. They recently raised the price to $99, which is not unreasonable given inflation and the rise of fuel and shipping costs. Unfortunately, the service has also degraded, and I am considering dropping it for that reason.

It really hit me this week. I ordered a bunch of loupes from eBay last weekend, as Schneider stopped manufacturing them 2 years ago and they are now officially listed as discontinued, and old-new stock of other reputable makers like Leica, Cabin/Mamiya or Rodenstock are starting to dwindle. At the same time, I ordered a few items using Amazon Prime (5 orders in all, 3 from Amazon themselves, 2 from third-party vendors but fulfilled by Amazon). All my eBay items have already arrived, including some shipped all the way from Canada that arrived yesterday, but only one of the Amazon items has arrived. Something is seriously wrong when Amazon’s vaunted logistics cannot match individual sellers on the fleabay.

The problem cannot be laid at the door of the shipping companies, the problem is that Amazon is taking longer to ship the items in the first place. It is an open question whether that delay is intentional (as seems to be the case for Amazon free super shipping orders, or when Netflix delays heavy customers’ DVDs to rate-limit them and thus reduce its shipping costs).

One other factor that has decreased the value of the service is the increasing proportion of items that are part of Amazon’s obnoxious add-on item program. Contrary to Amazon’s statements, many of the items downgraded to add-on status are not ones that were unavailable previously, but rather items that were previously eligible for Prime but no longer are. If I have to accumulate $25 in orders, I might as well go back to the free super saver shipping.

To sweeten the sticker shock, Amazon is bundling streaming video and music, and the Kindle lending library. Those services have essentially zero value for me, as the movie selection is as dismal as Netflix’s (mostly C-list or really old movies, hardly anything you might want to watch), streaming does not have good classical music coverage, and I refuse to use Kindle due to their predatory practices.

I find I am buying considerably less from Amazon these days:

  • Since they introduced sales tax, they are often not competitive with bricks-and-mortar retailers like Target (which will give you an extra 5% discount for using their REDcard) or B&H.
  • I refuse to buy books from Amazon (eBooks from iBooks or straight from the publisher like O’Reilly).
  • I buy my classical music from ArkivMusic (for CDs and SACDs, and they have their own $20/year equivalent of Prime) or FLAC sites like B&W Society of Sound, Linn, eClassical and Chandos.
  • I get my photo gear, computers and other electronics from B&H whenever possible, and that probably accounts for the bulk of my former Amazon dollars.

What’s left?

  • Oddball items hard to source otherwise
  • Tools
  • Household supplies (although I get most of these from Costco or, admittedly an Amazon company now).
  • Very occasionally some specialty grocery items and clothes.

I used Amazon’s handy order history export (temperamental, it fails if you have Amazon Honor System transactions in the selected date range) to calculate how much I spend with them (removing Adorama as they are a big outlier), and the trend is clearly unfavorable to Amazon since the high water mark of 2011. Their changes to Prime (pay more for worse service) are certainly not helping.


Externalities again

I just wasted half an hour of my life on the phone with my credit card company’s fraud department, as someone attempted to buy expensive tickets from an airline in Panama. Most likely my card number was compromised by Target, although it could also be due to Adobe.

It is actually surprising such breaches do not occur on a daily basis—the persons paying for the costs of a compromise (the card holder, defrauded merchants and their credit card companies via the cost of operating their fraud departments) are not the same as those paying for the security measures that would prevent the said breach, a textbook example of what economists call an externality. There are reputational costs to a business that has a major security breach, but they are occurring so often consumers are getting numbed to them.

Many states have mandatory breach disclosure laws, following California’s example. It is time for legislatures to take the next step and impose statutory damages for data breaches, e.g. $100 per compromised credit card number, $1000 per compromised social security number, and so on. In Target’s case, 40 million compromised credit cards multiplied by $100 would mean $4 billion in damages. That would make management take notice and stop paying mere lip service to security. It might also jump-start the long overdue migration to EMV chip-and-PIN cards in the United States.

The real electromagnetic emissions danger

I live 1.2km away from Sutro Tower in San Francisco. At my wife’s request I was trying to calculate the safe radius at which emissions from the transmitters at Sutro Tower are of the same power as a cell phone held a meter away, with back-of-the-envelope calculations using the inverse square law and Wikipedia’s table of radio powers.

I was shocked to find out the total power from the transmitters is about 8 megawatts, not in the kilowatt range I was expecting, and once reached 29MW. For comparison, the power of France’s first-generation PWR nuclear reactors is 900MW, and a typical cellular tower is 100W to 500W. If I use 2W as the reference, this yields a “safe” radius of 2km, which excludes many desirable San Francisco neighborhoods like Twin Peaks, Forest Hill or Noe Valley.

I looked up the most recent Environmental Impact Report following the DTV transition, and it mentions a FCC maximum allowed flux level of 0.2mW/cm2, and the measured levels in the Midtown Terrace neighborhood immediately adjacent to Sutro Tower reach 4% of this max level.

On further investigation, this is not one of those situations where US standards are significantly more lax than those in Europe, as France or the UK have the same level, derived from an international NGO called the ICNIRP. Interestingly, according to the WHO the maximum allowed emissions in such environmental paragons as Russia and China are one hundredth as high as those in the US or Europe and are just as science-based as those from ICNIRP (remember, for all its faults, the Soviet Union ranked very highly in maths and physics education & research, and in health care).

The ICNIRP/FCC standard is equivalent to a 25W isotropic emitter within a 1 meter radius, or 12x 2G GSM cell phones. Anyone who has experienced the squeal of unshielded and unpowered speakers next to an actively transmitting GSM phone will be skeptical about their claims that this is a safe level. Their methodology is based solely on the thermal effects of non-ionizing radiation, as if this were a mere microwave oven shielding exercise, and assumes that cells are otherwise unaffected by electromagnetism or cumulative exposure. This seems unwarrantedly optimistic.

People worry about cancer risks associated with radio frequency emissions from cell phone towers and cell phones themselves, but the real risk comes from overlooked obsolete technologies like TV and FM radio.

What to do? Getting a site survey from a Professional Engineer using calibrated equipment costs $1,500, which is something you would only do as part of a final inspection while buying a house. Most RF power meters sold on places like Amazon, usually in the $300 range, are pieces of junk with suggested applications like detecting paranormal activity and ghosts. Most likely solid engineering and metrology are optional given their application domain. Professional T&M gear like an Agilent V3500A or a Wandel & Goltermann/Narda EMR-300 cost $2,000 and $6,000 respectively, so the DIY route is also expensive.

Update (2014-03-08):

My father worked on some projects in the Soviet Union in the Seventies. He told me their workplace safety standards were much more stringent than the ones in the West. Workers were not allowed to lift weights above 25kg, for instance.

Afsheen’s mindset list

Beloit College is famous for its Mindset List, which explains to teachers the radically different world view students have, because their assumptions and experience are different. One example from this year’s list: “GM means food that is Genetically Modified”.

I tried to imagine what the list looks like when my daughter starts University.

Some are no-brainers, as they have already occurred:

  • A phone call has always involved both video and sound
  • A computing device is always touch-enabled

For some others, I may have to go out on a limb:

  • Cars have always been self-driving

Update (2015-08-25):

  • House roofs have always been tiled with solar panels

How the iPad Mini killed my iPhone

The single greatest feature of the iPad is the fact it cannot receive phone calls. Despite being a telecoms engineer by training, I despise phones, and it seems the millennial generation shares my disdain, as it favors less intrusive means of communication like texting.

The iPad is an essential device for me. I am on a 2-year upgrade cycle (at best) for phones, a 5-year cycle for my desktop Mac Pro, and have stopped using laptops altogether, but I will get every single iteration of the iPad. Now, even though my jacket has a pocket sized large enough to hold my full-sized iPad, the weight and bulk means I seldom did so, and kept it in my bag, which I rarely take out with me when going out for lunch. When I saw the iPad Mini and how lightweight it was, I bought one and started carrying it with me all the time.

The Mini is not a replacement for my Retina iPad, as my worsening eyesight makes it a strain for sustained reading, which is why I kept my grandfathered unlimited AT&T data plan on the full-sized iPad and got a limited Verizon plan on the Mini.

No, the device that was displaced is actually my iPhone. The iPad Mini weighs barely twice as much, is thinner, fits in my jacket pocket but has a screen 4 times the size while remaining single-hand-holdable, and is actually usable as a web browsing device or eBook reader, unlike the iPhone’s cramped screen. I don’t believe in the 5-inch phablet form factor, which combines the cramped screen of a phone with the the bulk of a tablet, i.e. the worst of both worlds. I find I never use the iPhone as anything else than a dumb phone any more. I consume less than 60 minutes of voice per month, and if my wife and my startup’s co-founder would let me, I would ditch mobile phones altogether.

Alas I am unable to cut the wireless phone tether, but there is no point in my spending $100 a month on an unlimited data plan for my Verizon iPhone 4, so now that my contract ended, I ported my number over to my old unlocked AT&T iPhone 3GS with a prepaid plan from Airvoice (a MVNO that has the cheapest rates I could find online). At $0.10 a minute without any exorbitant cellco taxes or spurious surcharges, I can expect to spend $6 a month, or 94% savings. That more than covers the $20 a month I pay extra for the iPad Mini’s data plan. The only reason I still use an iPhone instead of switching to a dumbphone is the automatic address book synchronization with my Mac and iOS devices.

Crime does not pay

Two years after the Staceycide, the spot is still vacant. At a reported rent of $65,000 a month, that adds up to a cool $1.8M loss for the greedy landlords who pushed them out of business.

Stacey's, RIP

It seems they finally found a new tenant: a CVS pharmacy occupies the premises now.

Deep packet inspection rears it ugly head

Last Friday I started noticing error messages in my production environment. URLs were being mangled, two consecutive characters being replaced by 0x80 and 0x01 or 0x80 and 0x04, causing UTF-8 decode exceptions to be logged, as well as failures for the cryptographic hash function we use to secure our URLs. As a general principle, I take any such unexpected exceptions very seriously and started investigating them, one concern being that some of our custom C extensions to nginx could be responsible for data corruption under heavy load.

I ran snoop (a Solaris utility similar to tcpdump) on one of our production servers, and after combing through 180MB of packet traces with Wireshark, it turned out the data was being corrupted before even hitting our web servers. While it was a relief to find out our own infrastructure was not to blame, I still had to identify the culprit, e.g. whether our hosting provider’s switches, firewalls or load-balancers were to blame.

TCP has built-in checksums, so a malfunctioning switch working at layers 1–3 would not cause this problem, a corrupted packet would be dropped and resent, with a slight hit on performance but no errors. Thus the problem would need to be at a L4 or higher device such as a load balancer.

I added some extra logging and let it run over the weekend. After analyzing the data, it turns out the problem is very circumscribed (76 requests out of hundreds of millions), and all the affected IP addresses come from the same ISP, Singapore Telecom Magix (AS9506). The only plausible explanation is that SingTel is running some sort of deep packet inspection gear, and some of the DPI gateways have corrupt memory or software bugs, that are causing the data flowing through them to get corrupted,

Deep Packet Inspection is a scourge the general public is insufficiently aware of. At a high level, DPI gateways watch over your shoulder as you use the Internet. They decode the data packets passing through them, reconstruct unencrypted HTTP requests (in other words, spy on your browsing history). In their transparent proxy incarnation, they can rewrite the requests or responses. Verizon Wireless uses the technology to resize and recompress images or videos requested by smartphones. Back when I used to work for France Telecom (circa 1996-1999), vendors would regularly approach us to peddle their wares and how they would allow us to price-gouge our customers more effectively. Hardware has progressed dramatically since and a single Xeon processor is capable of inspecting at least 10 Gbps of data.

The whole premise of DPI and other snooping devices is profoundly repugnant to me as a former network engineer, on both moral and technical grounds. Any additional “bump in the wire” slows things down and is yet another potential point of failure, as shown by this incident, but the potential for abuse is the real concern. Not to mince words, the legitimate purposes for the technology, such as fighting cybercrime, are just rationalizations, it was really developed for purposes most people would consider abusive.

When I joined FT, I had to go to a Paris courthouse and swear a solemn oath to defend the privacy of our customers’ communications, and report any infringement of the same. DPI technology originates in spy agencies, and is much beloved of authoritarian governments. China uses the technology, combined with voice recognition, to drop calls at the merest mention of the word “protest”. The Ben Ali regime in Tunisia used it to snoop Facebook users’ authentication cookies. Singapore’s government has a well-demonstrated intolerance of criticism, and who knows what SingTel is doing with their defective gear? Western companies like Cisco were disgracefully eager to sell censorware to dictatorships, but those governments now have homegrown capabilities from the likes of Huawei.

For telco oligopolies, the endgame is to practice perfect price discrimination, e.g. charge you more for packets that carry a voice over IP call or a Netflix video on demand session that compete with the carriers’ own services. Telcos and cablecos cannot be permitted to use their stranglehold over public networks for what is essentially racketeering. Strowger invented the automatic telephone switch because the operator at his manual exchange would divert his calls to one of his competitors, her husband. Telcos, in their monopolistic arrogance, feel a sense of entitlement to all the value the network creates, even when they are not responsible, and want to reverse this. Letting them get away with it, as is consistently the case in the US, is a recipe for long-term economic stagnation.

What can we as the general public do to fight back? The telcos are one of the largest lobbies in Washington, and wireless spectrum auction fees are one of the crutches propping up Western budgets, so help is unlikely to come from the venal legislatures. The most practical option is to start using SSL and DNSSEC for everything. Google now offers an encrypted search option and Facebook has an option to use SSL for the entire session, not just for login.

Update (2012-10-16):

It seems Verizon also uses DPI to build marketing profiles on its users, i.e. categorizes you based on your browsing history and sells you to marketers. You can opt out, but the practice is deeply worrisome.

Hey Apple…

Some improvements you should consider:

  • Sync iPods, iPhones and iPads over WiFi. Cables are so twentieth century. Palm had bluetooth sync working ten years ago, and 802.11n has the same real-world speed as USB. You could then simply extend this to sync the device to the cloud instead of a specific computer.
  • Ditching DVDs to offer an OS reinstall USB flash drive on the new MacBook Airs and Pros is a good idea, but the stick is easy to misplace. How about soldering a read-only USB drive directly onto the motherboard so it can never be lost?
  • When someone enters an address in a Calendar entry on iOS, make it clickable and linked to the Maps app, the way addresses in Contacts are. Copying and pasting them manually is a drag.
  • Stop adding useless frills like “stationery” to, and make the default chronological sort order switchable to “most recent on top”.
  • Add HDMI CEC support to the AppleTV. It would be nice to have a HDTV automatically switch over to the AppleTV’s HDMI input when you try to access it. Speaking of which, it would be nice to have an option to disable the audio out on HDMI, e.g. if you have a decent surround sound system connected to it over Toslink and don’t want the TV’s tinny speakers to kick in.

I love my ISP

Not only do Webpass give me fast 45Mbps symmetrical access for $45/month, with no capricious restrictions or anticompetitive shenanigans, but they are also real mensches.

Putting customers first

When you visit the Dell website, the first thing they force you to decide is whether you are a Home, Small Business or “Enterprise” business customer. At one point, the thin and light laptops were only available in the Enterprise section—perhaps plebs and small businesses are judged unworthy of appreciating the finer things in life, unlike the kleptocrats who run large corporations. We hoi polloi presumably should be content with our fate and make do with last year’s (decade’s?) technology.

When you search for products on Amazon, you have to select a “Department” to enable sorting by price. What do I care whether a microfiber cloth was filed under “Automotive” or “Electronics”? Taxonomies are inherently subjective, a fact librarians know well, but is surprisingly poorly understood outside the field.

Both cases illustrate what happens when a self-centered organization puts its internal structure and implementation details ahead of its customers.

Dear Parallels

Since you keep hitting me with these spammy popups no matter how many times I click on “Do not show again”, you leave me no choice but to switch to VirtualBox (much better software in any case, and less Windows integration means less chances a virus breaking out of the virtualized Windows ghetto.

Oh, and installing MacFuse without asking permission (unlike VMware Fusion): not cool.

Don’t let the door hit you on the way out.

Will Adobe ever learn?

In a triumph of hope over experience, I recently “upgraded” from Adobe CS3 Design Standard to CS5 Design Standard. I hardly ever use Photoshop any more since I started using Aperture and Lightroom (originally a Macromedia product, no matter what the lame “Adobe Photoshop Lightroom” face-saving branding may try to claim), the main driver for the purchase was actually InDesign CS5 and its ePub functions.

Of course, this is Adobe. Previous versions gratuitously included crud like a full Opera install (an older version, insecure, naturally) just to display a splash screen, or a full MySQL install to power Acrobat search. I never install Acrobat, of course, since that bloated and bug-ridden piece of garbage managed to steal the crown for most insecure software from Internet Explorer, no small feat.

Adobe does not want to confuse users with streamlined and efficient software, so they decided to include the mostly useless Growl on-screen notification program to nag you into registering. Increasing bloat and attack surface for malware is not a good idea, nor is interrupting creative people’s flow with interruptions. Of course, helping clients Get Things Done is a low priority at Adobe, as evidenced by their product choices.

You have to pity the Growl developers, whose reputation will suffer from guilt by association. I dislike interruptions and do not find it useful, but many people do and rave about it. They installed it by choice, not as a sneaky drive-by install for slimy marketing purposes.

Some more annoyances in CS5:

  • The pricing for the suite is more than the sum of its parts: $200 each for Photoshop, InDesign or Illustrator, $700 for Design Standard. I suppose they must think Acrobat and their online tie-ins have a value of $100 (hint: they forgot the negative sign).
  • Of course, they won’t let you upgrade individual component applications.
  • On the plus side, they now have the decency to include Acrobat on a separate CD, so you can discard it immediately and not risk installing it as a side-effect of installing the apps that are actually useful.
  • The icons were designed by the world’s laziest and most creatively bankrupt designer, just as with CS3 and CS4
  • Performance on a high-end 8-core or 12-core Mac is actually slower than on a lower-end configuration, thanks to legacy cruft and incompetence.
  • It is slower to load on my wife’s MacBook Pro. Each successive version of OS X is faster on the same hardware, Microsoft and Adobe deliver software that gets progressively slower.

In other words, unlike Lightroom, CS5 is designed to be endured, not to delight.

Incensed at Mozilla

One of the greatest features in the Webkit-based browsers (Apple’s Safari and Google Chrome) is WebSQLdatabase, the ability for a web site to store information in a SQLite database on your browser accessible via JavaScript. This allows web developers to build database-enabled applications that run entirely in the browser, without requiring a server. This is very useful for mobile devices, which in the US enjoy flaky network connectivity at best. One very handsome example is the iPad-optimized Every Time Zone webapp.

SQLite is probably the most important open-source project you have never heard of. It is a simple, streamlined and efficient embedded database. Firefox stores its bookmarks in it. Google distributes its database of phishing sites in that format. Sun’s industrial-strength Solaris operating system stores the list of services it runs on boot in it—if it were to fail, a server would be crippled so that is a pretty strong vote of confidence. Adobe Lightroom and Apple’s Aperture use it to store their database, as do most Mac applications that use the CoreData framework, and many iPhone apps. In other words, it is robust and proven mission-critical software that is widely yet invisibly deployed.

WebSQLdatabase basically makes the power of SQLite available to web developers trying to build apps that work offline, specially on mobile devices. No good deed goes unpunished, and the Mozilla foundation teamed up with unlikely bedfellow Microsoft to torpedo formal adoption of WebSQLdatabase as a web standard, on spurious grounds, and pushed an alternate standard called IndexedDB instead. To quote the Chromium team:

Q: Why this over WebSQLDatabase?

A: Microsoft and Mozilla have made it very clear they will not implement SQL in the browser.  If you want to argue this is silly, talk to them, not me.

IndexedDB is several steps backwards. Instead of using powerful, expressive and mature SQL technology, it uses a verbose JavaScript B-tree API that is a throwback to the 1960s bad old days of hierarchical databases and ISAM, requires a lot more work from the developer, for no good reason. To add injury to insult, Firefox 4’s implementation of IndexedDB is actually built on top of SQLite. The end result will be that web developers will need to build a SQL emulation library on top of IndexedDB to restore the SQLite functionality deliberately crippled by IndexedDB. If there is one constant in software engineering, it is that multiple layers add brittleness and impair performance.

Of course, both Mozilla and Microsoft are irrelevant on mobiles, where WebKit has essentially won the day, so why should this matter? Microsoft has always been a hindrance to the development of the web, since they have to protect the Windows API from competition by increasingly capable webapps, but I cannot understand Mozilla’s attitude, except possibly knee-jerk not-invented-here syndrome and petulance at being upstaged by WebKit. WebSQLdatabase is not perfect—to reach its full potential, it needs and automatic replication and sync facility between the local database and the website’s own database, but it is light years ahead of IndexedDB in terms of power and productivity.

I am so irritated by Mozilla’s attitude that after 10 years of using Mozilla-based browsers, I switched today from Firefox to Chrome as my primary browser. Migrating was surprisingly easy. Key functionality like bookmark keywords, AdBlock, FlashBlock, a developer console, and the ability to whitelist domains for cookies, all have equivalents on Chrome. The main regressions are bookmark tags, and Chrome’s sync options are not yet equivalent to Weave‘s. At some point I will need to roll my own password syncing facility (Chrome stores its passwords in the OS X keychain, which is also used by Safari and Camino).

Forbidden functions

When I first read Asimov’s Foundation as a child in 1980 or so, I was blown away by the idea that a mathematical operation could be forbidden.

“Before you are done with me, young man, you will learn to apply psychohistory to all problems as a matter of course. – Observe.” Seldon removed his calculator pad from the pouch at his belt. Men said he kept one beneath his pillow for use in moments of wakefulness. Its gray, glossy finish was slightly worn by use. Seldon’s nimble fingers, spotted now with age, played along the files and rows of buttons that filled its surface. Red symbols glowed out from the upper tier.

He said, “That represents the condition of the Empire at present.”

He waited.

Gaal said finally, “Surely that is not a complete representation.”

“No, not complete,” said Seldon. “I am glad you do not accept my word blindly. However, this is an approximation which will serve to demonstrate the proposition. Will you accept that?”

“Subject to my later verification of the derivation of the function, yes.” Gaal was carefully avoiding a possible trap.

“Good. Add to this the known probability of Imperial assassination, viceregal revolt, the contemporary recurrence of periods of economic depression, the declining rate of planetary explorations, the…”

He proceeded. As each item was mentioned, new symbols sprang to life at his touch, and melted into the basic function which expanded and changed.

Gaal stopped him only once. “I don’t see the validity of that set-transformation.”

Seldon repeated it more slowly.

Gaal said, “But that is done by way of a forbidden sociooperation.”

“Good. You are quick, but not yet quick enough. It is not forbidden in this connection. Let me do it by expansions.”

Isaac Asimov, Foundation, Chapter 4 (emphasis mine)

Later, I learned in a Byte article on Karmarkar’s algorithm for linear programming that AT&T had patented it. The idea that mathematical algorithms are patentable was just as absurd for a 17 year old. Not quite as absurd as patenting living organisms’ genome still seems to me, but close. Forbidden mathematics seemed like something from the Middle Ages, or Stalinist Russia.

Of course, this is exactly what happens when our governments try to outlaw cryptography, or the media industry tries to ban algorithms like DeCSS, or even public discussion by academics of flaws in their poorly designed cryptosystems. There is an apocryphal myth that Pythagoras’ mathemato-mystical cult tried to assassinate the man who first proved the square root of two is an irrational number. Mathematics can be an inconvenient truth at times.

RapidSSL 1 – GoDaddy 0

My new company’s website uses SSL. I ordered an “extended validation” certificate from GoDaddy, instead of my usual CA, RapidSSL/GeoTrust, because GoDaddy’s EV certificates were cheap. EV certificates are security theater more than anything else, I probably should not have bothered.

Immediately after switching from my earlier “snake oil” self-signed test certificate to the production certificate, I saw SSL errors on Google Chrome for Mac and Safari for Mac, i.e. the two browsers that use OS X’s built-in crypto and certificate store. I suppose I should have tested the certificate on another server before going live, but I trusted GoDaddy (they are my DNS registrars, and competent, if garish).

Big mistake.

I called their tech support hotline, which is incredibly grating because of the verbose phone tree that keeps trying to push add-ons (I guess it is consistent with the monstrosity that is their home page).

After a while, I got a first-level tech. He asked whether I saw the certificate error on Google Chrome for Windows. At that point, I was irate enough to use a four-letter word. Our customers are Android mobile app developers. A significant chunk of them use Macs, and almost none (less than 5%) use IE, so know-nothing “All the world is IE” demographics are not exactly applicable.

After about half an hour of getting the run-around and escalating to level 2, with my business partner Michael getting progressively more anxious in the background, the level 1 CSR tells me the level 2 one can’t reproduce the problem (I reproduced it on three different Macs in two different locations). I gave them an ultimatum: fix it within 10 minutes or I would switch. At this point, the L1 CSR told me he had exhausted all his options, but I could call their “RA” department, and offered to switch me. Inevitably, the call transfer failed.

I dialed their SSL number, and in parallel started the certificate application process on RapidSSL. They offered a free competitive upgrade, I tried it, and within 3 minutes I had my fresh new, and functional certificate, valid for 3 years, all for free and in less time than it takes to listen to GoDaddy’s obnoxious phone tree (all about “we pride ourselves in customer service” and other Orwellian corporate babble).

I then called GoDaddy’s billing department to get a refund. Surprisingly, the process was very fast and smooth. I guess it is well-trod.

The moral of the story: GoDaddy—bad. RapidSSL—good.

Update (2012-08-26)

I switched my DNS business from GoDaddy to in December 2011 after Bob Parsons’ despicable elephant-hunting stunt.

What is heard, and what is not heard

French economist Frédéric Bastiat (1801–1850) wrote a pamphlet titled Ce qu’on voit et ce qu’on ne voit pas (What is Seen and What is Not Seen) where he demolishes the make-work fallacy in economics. When Jacques Bonhomme’s child breaks his window, paying for a replacement will circulate money in the economy, and stimulate the glassmakers’ trade. This is the visible effect. Bastiat urges us to consider what is not seen, i.e. opportunity costs, such as other, more productive uses for the money that are forgone due to the unexpected expense. This lesson is still relevant. The cost of repairing New Orleans after Katrina, or cleaning the Gulf after Deepwater Horizon, will cause a temporary boost in GDP statistics, but this is illusory and undesirable, another example of how poorly conceived metrics can distort thinking.

Another example is that of electric cars. Advocates for the blind have raised a ruckus about the dangers to blind people from quiet electric cars they cannot hear or dodge. Nissan just announced that their Leaf electric car will include a speaker and deliberately generate noise, in part to comply with the Japanese Transport Ministry’s requirements. To add injury to insult, the sound selected is apparently a sweeping sine wave, a type of sound that is incredibly grating compared to more natural sounds, including that of machinery.

Unfortunately, this is illustrates the fallacy Bastiat pointed out. Authorities are focusing on the visible (well, inaudible) first-order effect, but what is not seen matters as much. Most urban noise stems from transportation, and that noise pollution has major adverse impact on stress levels, sleep hygiene, and causes high blood pressure and cardiac problems from children to adults to the elderly. According to the WHO, for 2006 in the UK alone, an estimated 3,000+ deaths due to heart attacks can be attributed to noise pollution (out of 100,000+).

These figures are mind-boggling. For a country the size of the US, that probably comes around to five  or six 9/11 death tolls per year. Quiet electric cars should be hailed as a blessing, not a danger. There are other ways to address the legitimate concerns of the blind, e.g. by mandating transponders on cars and providing receivers for the blind.