IT

Scanner group test

TL:DR—avoid scanners with Contact Image Sensors if you care at all about color fidelity.

Vermeer it is not

After my abortive trial of the Colortrac SmartLF Scan, I did a comparative test of scanning one of my daughter’s A3-sized drawings on a number of scanners I had handy.

Scanner Sensor Scan
Colortrac SmartLF Scan CIS ScanLF.jpg
Epson Perfection Photo V500 Photo (manually stitched) CCD Epson_V500.jpeg
Epson Perfection V19 (manually stitched) CIS Epson_V19.jpg
Fujitsu ScanSnap S1500M (using a carrier sheet and the built-in stitching) CCD S1500M_carriersheet.jpg
Fujitsu ScanSnap SV600 CCD SV600.jpg
Fuji X-Pro2 with XF 35mm f/1.4 lens, mounted on a Kaiser RS2 XA copy stand with IKEA KVART 3-spot floor lamp (CCT 2800K, a mediocre 82 CRI as measured with my UPRtek CV600) CMOS X-Pro2.jpg

I was shocked by the wide variance in the results, as was my wife. This is most obvious in the orange flower on the right.

Comparison

I scanned a swatch of the orange using a Nix Pro Color Sensor (it’s the orange square in the upper right corner of each scan in the comparison above). When viewed on my freshly calibrated NEC PA302W SpectraView II monitor, the Epson V500 scan is closest, followed by the ScanSnap SV600.

The two scanners using Contact Image Sensor (CIS) technology yielded dismal results. CIS are used in low-end scanners, and they have the benefit of low power usage, which is why the only USB bus-powered scanners available are all CIS models. CIS sensors begat the CMOS sensors used by the vast majority of digital cameras today, superseding CCDs in that application, I would not have expected such a gap in quality.

The digital camera scan was also quite disappointing. I blame the poor quality of the LEDs in the IKEA KVART three-headed lamp I used (pro tip: avoid IKEA LEDs like the plague, they are uniformly horrendous).

I was pleasantly surprised by the excellent performance of the S1500M document scanner. It is meant to be used for scanning sheaves of documents, not artwork, but Fujitsu did not skimp and used a CCD sensor element, and it shows.

Pro tip: a piece of anti-reflective Museum Glass or equivalent can help with curled originals on the ScanSnap SV600. I got mine from scraps at a framing shop. I can’t see a trace of reflections on the scan, unlike on the copy stand.

Avery 22807 template for InDesign

The Avery 22807 2-inch circular stickers are a good alternative to Moo, PSPrint et al when you need a small quantity of stickers in a hurry. Unfortunately Avery has not seen it fit to provide usable InDesign templates as they do with some of their other sticker SKUs, only Microsoft Word, which is needless to say inadequate. A search for “Avery 22807 Indesign template” yielded some, but they have issues with missing linked PDF files.

I reverse-engineered the Microsoft template to build one of my own, with dimensions (including the tricky almost-but-not-quite square grid spaced at 5/8″ horizontally but 7/12″ vertically) to simplify “Step and Repeat…”.

I have only tested this with my InDesign CS6, not sure if it will work with older versions.

Avery 22807 2-inch circular labels.indt

Avoiding counterfeit goods on Amazon: mission impossible?

I mentioned previously that I seldom shop for electronics on Amazon.com any more, preferring B&H Photo whenever possible. I now have another reason: avoiding counterfeit goods.

My company boardroom is in an electromagnetic war zone—dozens of competing WiFi access points combined with electronic interference from the US-101 highway just outside make WiFi reception tenuous at best, and unusable more often than not. To work around this, we set up a wired Ethernet switch, and since most of our staff use MacBook Airs, Apple USB Ethernet adapters purchased from Amazon. When I side-graded from my 15″ Retina MacBook Pro to a much more portable 12″ Retina MacBook, I wasn’t able to connect using the dongle, and the name of the device was interspersed with Chinese characters. At first I thought it was an issue with my Satechi USB-C hub, but I experienced the same problems via a genuine Apple USB-C multiport adapter as well.

Eventually I figured out the Ethernet dongles were counterfeit. The packaging, while very similar to Apple’s, was just a tiny bit off, like amateurish margins between the Apple logo and the edges of the card. On the dongles themselves, the side regulatory disclosures sticker was inset, not flush with the body of the adapter.

Counterfeiting is a major problem. By some accounts, one third of all Sandisk memory cards worldwide are counterfeits. In some cases like chargers or batteries, your equipment could be at risk, or even your very life. The counterfeit adapters we purchased from Amazon did not come from Amazon themselves but from a third-party merchant participating in the Amazon marketplace. To Amazon’s credit, we returned them for a prompt, no questions asked refund even though we bought them over six months ago, but it is hard to believe Amazon is unaware of the problem rather than willfully turning a blind eye to it.

My first reaction was to tell our Office Manager to make sure to buy only from Amazon rather than third-party merchants (pro tip: including “amazon” in your Amazon search terms will do that in most cases). Unfortunately, that may not be enough. Amazon has a “fulfilled by Amazon” program for merchants where you ship your goods to them, and they handle warehousing and fulfillment. These “fulfilled by Amazon” items are also more attractive to Prime members. One option Amazon offers is Stickerless, commingled inventory where the items you send are put into a common bin. Amazon still has the ability to trace the provenance of the item through its inventory management, but for purposes of order fulfillment they will be handled just like Amazon’s own stock. Some categories like groceries and beauty products are excluded, but electronics are not.

The implications are huge: even if the vendor is Amazon itself, you cannot be sure that the item is not counterfeit. All the more reason to buy only from trustworthy, single-vendor sites like B&H, even if shipping is a bit slower.

Chrome and AES-256 security: it’s not me, it’s you

This blog now supports the HTTP/2 protocol, courtesy of nginx 1.9.5 (PDF).

In the process, I was stymied by an “ERR_SPDY_INADEQUATE_TRANSPORT_SECURITY” error from Google Chrome. HTTP/2 mandates TLS de facto, if not in the strict letter of the specification, and it also forbids a number of obsolete or weaker SSL/TLS ciphers to only permit ones that are truly secure. After some considerable digging, I found out the issue is Google Chrome on Mac and Android (presumably Windows as well) does not support 256-bit AES in HTTP/2, and my server was set up to only accept 256-bit encryption (only the best will do for my readers!). The error message was misleading: it’s not the server but Chrome’s crypto which is lacking.

It seems the cryptographers at Google feel 128-bit AES in Galois Counter Mode is good enough, and they did not want to be too far apart from Firefox (which does not support it either, and just fails without even the courtesy of an error message). In contrast, Safari on Yosemite supports AES-256-CBC (not ideal, I know, but that’s also what Chrome supports if HTTP/2 is turned off) and AES-256-GCM on El Capitan and iOS 9. Here are the settings your browser uses:

This is disappointing. AES-256-GCM is supported in hardware on most Intel hardware nowadays (all but lowest-end chips have the AES-NI instructions) and in the ARMv8-A architecture supported by most smartphones and mobile devices today, where the extra CPU load would matter most. I wonder how much of this is driven by Google’s fondness for Dan Bernstein’s ChaCha20+Poly1305 algorithms. Excellent as they may be, they are not implemented in hardware on the most common platforms, nor implemented at all in OpenSSL. It is quite disconcerting that my phone has better crypto than my desktop browser.

I ended up resolving the issue by loosening my cipher list from AES256+EECDH to EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH, but Chrome really should catch up and not let itself be hobbled by the increasingly irrelevant Firefox and its hoary NSS crypto.

I probably sound harsher than I intended towards the Google crypto team. The backward compatibility issues they have to deal with, from poorly designed TLS standards to broken web server software, intrusive anti-virus or corporate proxy servers mean a lot of their energy goes into exception cases, rather than implementing the latest and greatest in crypto algorithms.

Update (2017-01-18):

It looks like Chrome silently added AES256-GCM support last year, as it now negotiates the ECDHE-RSA-AES256-GCM-SHA384 cipher on aes256gcm.majid.org.

How the iPad Mini killed my iPhone

The single greatest feature of the iPad is the fact it cannot receive phone calls. Despite being a telecoms engineer by training, I despise phones, and it seems the millennial generation shares my disdain, as it favors less intrusive means of communication like texting.

The iPad is an essential device for me. I am on a 2-year upgrade cycle (at best) for phones, a 5-year cycle for my desktop Mac Pro, and have stopped using laptops altogether, but I will get every single iteration of the iPad. Now, even though my jacket has a pocket sized large enough to hold my full-sized iPad, the weight and bulk means I seldom did so, and kept it in my bag, which I rarely take out with me when going out for lunch. When I saw the iPad Mini and how lightweight it was, I bought one and started carrying it with me all the time.

The Mini is not a replacement for my Retina iPad, as my worsening eyesight makes it a strain for sustained reading, which is why I kept my grandfathered unlimited AT&T data plan on the full-sized iPad and got a limited Verizon plan on the Mini.

No, the device that was displaced is actually my iPhone. The iPad Mini weighs barely twice as much, is thinner, fits in my jacket pocket but has a screen 4 times the size while remaining single-hand-holdable, and is actually usable as a web browsing device or eBook reader, unlike the iPhone’s cramped screen. I don’t believe in the 5-inch phablet form factor, which combines the cramped screen of a phone with the the bulk of a tablet, i.e. the worst of both worlds. I find I never use the iPhone as anything else than a dumb phone any more. I consume less than 60 minutes of voice per month, and if my wife and my startup’s co-founder would let me, I would ditch mobile phones altogether.

Alas I am unable to cut the wireless phone tether, but there is no point in my spending $100 a month on an unlimited data plan for my Verizon iPhone 4, so now that my contract ended, I ported my number over to my old unlocked AT&T iPhone 3GS with a prepaid plan from Airvoice (a MVNO that has the cheapest rates I could find online). At $0.10 a minute without any exorbitant cellco taxes or spurious surcharges, I can expect to spend $6 a month, or 94% savings. That more than covers the $20 a month I pay extra for the iPad Mini’s data plan. The only reason I still use an iPhone instead of switching to a dumbphone is the automatic address book synchronization with my Mac and iOS devices.

If WordPress updates hang on a 64-bit OS

The WordPress instance running this site was no longer able to automatically update plugins (and presumably not the core either) after I upgraded from a 32-bit to a sparkling fresh 64-bit PHP install at Joyent. It would start the update, and show a spinning logo and then just hang.

After much debugging, I found out the problem is that the class-pclzip.php that is responsible for unzipping was failing silently with the message:

Downloading update from http://downloads.wordpress.org/plugin/yet-another-related-posts-plugin.3.5.2.zip

Unpacking the update…

Abort class-pclzip.php : Missing zlib extensions

This isn’t terribly helpful, but digging in, it turns out that class depends on the PHP zlib module, and on 64-bit operating systems (more precisely, operating systems with 64-bit large file support enabled), zlib.h #defines gzopen to be gzopen64. PHP does not protect itself adequately and thus the PHP function gzopen gets renamed gzopen64 as well, this throwing class-pclzip.php for a loop, along with a number of other systems like PEAR.

Fixing this requires recompiling PHP. Ubuntu Karmic includes a work-around, but I run Solaris and build from source, so I contributed a patch filed under bug #53829.

Automattic should probably patch class-pclzip.php to deal with gzopen/gzopen64 as there are a great many broken PHP installs out there (the PHP bug has been open for over a year and a half without what I would consider an acceptable solution), and it is surprisingly difficult to find a solution online. I guess a great many WordPress installs are still 32-bit, which is kind of sad.

Scanning your iTunes library for DRM-infested books

Tor, the leading publisher for Science Fiction and Fantasy books, announced they would be doing away with DRM in their eBooks. The product pages for their books on iBooks now mention “At the publisher’s request, this title is being sold without Digital Rights Management software (DRM) applied”. I figured it would be a good idea to uncripple the many Tor eBooks I have in my collection.

I wrote a quick little Python script to scan my growing iBooks library for books that could be updated. The procedure is to delete the book from both iTunes and iPads, then download it anew (restarting iTunes is also needed after deleting). Apple keeps track of your purchases and will not charge you again.

#!/usr/bin/env python
import sys, os.path, glob, zipfile, platform, xml.etree.ElementTree

# publishers who have forsaken DRM
good = ['Tom Doherty']

if platform.mac_ver()[0] > '10.8':
  bookdir = os.path.expanduser('~/Library/Containers/com.apple.BKAgentService/Data/Documents/iBooks')
else:
  bookdir = os.path.expanduser('~/Music/iTunes/iTunes Music/Books')

os.chdir(bookdir)

ok =  '\033[1;32mDRM-free    \033[0m'
bad = '\033[1;31mDRM-infested\033[0m'

count = 0
salvageable = 0

def extract(meta):
  creator = ''
  status = ''
  pub = ''
  et = xml.etree.ElementTree.fromstring(meta)
  try:
    creator = et.findall('*{http://purl.org/dc/elements/1.1/}creator')
    creator = creator[0].text
    title = et.findall('*{http://purl.org/dc/elements/1.1/}title')
    title = title[0].text
  except:
    assert '!DOCTYPE plist' in meta
    next_tag = None
    for e in et[0].iter():
      if e.tag == 'key' and e.text in ('artistName', 'itemName'):
        next_tag = e.text
        continue
      if next_tag == 'artistName':
        creator = e.text
        next_tag = None
        continue
      elif next_tag == 'itemName':
        title = e.text
        next_tag = None
        continue
  pub = [x for x in good if x in meta]
  return creator, title, pub

def find_meta(file_list, opener):
  for m in file_list:
    if m.endswith('.opf') or m == 'iTunesMetadata.plist':
      meta = opener(m).read()
      return extract(meta)
  
for fn in glob.glob('*/*.epub'):
  status = ok
  suffix = ''
  if os.path.isdir(fn):
    suffix = '(directory)'
    if os.path.exists(fn + '/META-INF/encryption.xml'):
      status = bad
      count += 1
    meta = find_meta(os.listdir(fn), lambda x: open(fn + '/' + x))
  else:
    z = zipfile.ZipFile(fn)
    try:
      i = z.getinfo('META-INF/encryption.xml')
      status = bad
    except KeyError:
      pass
    meta = find_meta(z.namelist(), z.open)
    z.close()
  creator, title, pub = meta
  print status, fn, suffix
  print '\t', creator
  print '\t', title
  if status == bad and pub:
    print '\t\033[1;32mThis is published by', pub[0],
    print 'and could be re-downloaded DRM-free\033[0m'
    salvageable += 1

print count, 'books are DRM-infested'
print salvageable, 'could be cured'

Unfortunately, it seems like the DRM-stripping is still work in progress. Out of the Wheel of Time series, for instance, only the first one is now DRM-free on the iBooks store.

undr ~>drmbooks.py
DRM-free     Books/0083D0AEC37E08453347DD12B1C6F980.epub
    Greg Bear
    Blood Music
DRM-free     Books/09178837756A4DFF8347EC377345A37B.epub
    Heinz Wittenbrink
    RSS and Atom
DRM-free     Books/0AD752E995042C7E12F11917AB58C6B8.epub
    Wes McKinney
    Python for Data Analysis
DRM-free     Books/14BDC66A99E878EC232FFAFA73B341EF.epub
    Fritz Leiber
    Swords and Deviltry-Fafhrd and the Gray Mouser-Book1
DRM-free     Books/15A1D7FE9B7D815C6FBE1A9A77D7143E.epub
    Glen Cook
    A Fortress in Shadow
DRM-free     Books/1793F9DE1319B96FDE7E36EB8A1BC961.epub
    Scalzi, John
    Old Man’s War
DRM-free     Books/1D08BE221E8BC8F2A371EFEDE55029AC.epub
    Ben Fry
    Visualizing Data
DRM-free     Books/24D6EC36CDEA0C1E8612CC61A89EA098.epub
    None
    Node Cookbook
DRM-free     Books/29DA285F0051C431BD8BA3D1AEC5EAA6.epub
    Fritz Leiber
    The Swords of Lankhmar: Fafhrd and the Gray Mouser-Book 5
DRM-free     Books/2E88CD68DFD8408CD0E7C0ACB1E78714.epub
    Glen Cook
    A Cruel Wind: A Chronicle of the Dread Empire
DRM-free     Books/32996A9995040064818BAE4DFB66E92F.epub
    Kelly Link
    Magic for Beginners
DRM-free     Books/34D3CD13D47E5FEBC6DCF7EF011113BD.epub
    David Drake
    Lord of the Isles
DRM-infested Books/357298432.epub
    Iain M. Banks
    The Player of Games
DRM-infested Books/357311036.epub
    Iain M. Banks
    Use of Weapons
DRM-infested Books/357377857.epub
    Iain M. Banks
    Against a Dark Background
DRM-infested Books/357396585.epub
    Brent Weeks
    Night Angel: The Complete Trilogy
DRM-infested Books/357657026.epub
    Iain M. Banks
    Transition
DRM-infested Books/357658374.epub
    Po Bronson
    NurtureShock: New Thinking About Children
DRM-infested Books/357662058.epub
    Iain M. Banks
    Consider Phlebas
DRM-infested Books/357669769.epub
    Iain M. Banks
    Matter
DRM-infested Books/357914731.epub
    Herbert, Frank
    Dune Messiah
DRM-infested Books/357918110.epub
    Dalrymple, William
    City of Djinns
DRM-infested Books/357923567.epub
    Patrick Rothfuss
    The Name of the Wind
DRM-infested Books/357929995.epub
    Herbert, Frank
    Dune (40th Anniversary Edition)
DRM-infested Books/357969577.epub
    Herbert, Frank
    God Emperor of Dune
DRM-infested Books/357987322.epub
    Herbert, Frank
    Children of Dune
DRM-infested Books/357994537.epub
    Herbert, Frank
    Heretics of Dune
DRM-infested Books/357994652.epub
    William Dalrymple
    White Mughals: Love and Betrayal in Eighteenth-Century India
DRM-infested Books/357996119.epub
    Stross, Charles
    Wireless
DRM-infested Books/360601506.epub
    Ursula K. Le Guin
    The Dispossessed
DRM-infested Books/360609519.epub
    Greg Egan
    Schild’s Ladder
DRM-infested Books/360627712.epub
    Raymond E. Feist
    Rides a Dread Legion
DRM-infested Books/360627930.epub
    Neal Stephenson
    Anathem
DRM-infested Books/360628773.epub
    Raymond E. Feist
    At the Gates of Darkness
DRM-infested Books/360641088.epub
    Mihaly Csikszentmihalyi
    Flow
DRM-free     Books/361491495.epub
    Basil Hall Chamberlain
    Aino Folk-Tales
DRM-free     Books/361494664.epub
    Poul William Anderson
    Industrial Revolution
DRM-free     Books/361523763.epub
    Lafcadio Hearn
    The Romance of the Milky Way / And Other Studies & Stories
DRM-free     Books/361527545.epub
    Saki
    When William Came
DRM-free     Books/361539032.epub
    Saki
    The Chronicles of Clovis
DRM-free     Books/361557387.epub
    Saki
    Reginald in Russia and other sketches
DRM-free     Books/361557834.epub
    Sir Arthur Conan Doyle
    The Adventure of the Dying Detective
DRM-free     Books/361559391.epub
    Sir Arthur Conan Doyle
    The Valley of Fear
DRM-free     Books/361560694.epub
    Lafcadio Hearn
    Chita: a Memory of Last Island
DRM-free     Books/361561399.epub
    Confucius
    The Analects of Confucius (from the Chinese Classics)
DRM-free     Books/361562678.epub
    Saki
    The Toys of Peace, and other papers
DRM-free     Books/361562764.epub
    Sir Arthur Conan Doyle
    The Memoirs of Sherlock Holmes
DRM-free     Books/361564075.epub
    Poul William Anderson
    The Burning Bridge
DRM-free     Books/361564898.epub
    Henry David Thoreau
    Walden
DRM-free     Books/361565201.epub
    Saki
    Beasts and Super-Beasts
DRM-free     Books/361565806.epub
    Isaac Asimov
    Youth
DRM-free     Books/361572327.epub
    Lafcadio Hearn
    Kokoro / Japanese Inner Life Hints
DRM-free     Books/361573126.epub
    Sir Arthur Conan Doyle
    Through the Magic Door
DRM-free     Books/361575882.epub
    Sir Arthur Conan Doyle
    Tales of Terror and Mystery
DRM-free     Books/361578744.epub
    Lafcadio Hearn
    In Ghostly Japan
DRM-free     Books/361588265.epub
    Lafcadio Hearn
    Books and Habits from the Lectures of Lafcadio Hearn
DRM-free     Books/361673695.epub
    E. C. Babbitt
    More Jataka Tales
DRM-free     Books/361686559.epub
    Poul William Anderson
    Security
DRM-free     Books/361713863.epub
    Sir Arthur Conan Doyle
    The Return of Sherlock Holmes
DRM-free     Books/361721797.epub
    Sir Arthur Conan Doyle
    The Adventure of the Cardboard Box
DRM-free     Books/361725352.epub
    Saki
    The Unbearable Bassington
DRM-free     Books/361725959.epub
    Sir Arthur Conan Doyle
    The Adventure of Wisteria Lodge
DRM-free     Books/361726975.epub
    Saki
    Reginald
DRM-free     Books/361727237.epub
    Sir Arthur Conan Doyle
    The Adventure of the Red Circle
DRM-free     Books/361732286.epub
    Poul William Anderson
    The Valor of Cappen Varra
DRM-free     Books/361736043.epub
    Lafcadio Hearn
    Kwaidan: Stories and Studies of Strange Things
DRM-free     Books/361741563.epub
    Lafcadio Hearn
    Japan: an Attempt at Interpretation
DRM-free     Books/361743007.epub
    Ambrose Bierce
    The Devil’s Dictionary
DRM-free     Books/361743953.epub
    Sir Arthur Conan Doyle
    The Adventure of the Devil’s Foot
DRM-free     Books/361744178.epub
    Sir Arthur Conan Doyle
    His Last Bow
DRM-free     Books/361745602.epub
    Poul William Anderson
    The Sensitive Man
DRM-infested Books/362435686.epub
    Ansary, Tamim
    Destiny Disrupted
DRM-infested Books/366773380.epub
    Esslemont, Ian C. C.
    Return of the Crimson Guard
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/373338999.epub
    Jordan, Robert
    The Path of Daggers
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-free     Books/375554215.epub
    Orson Scott Card
    The Lost Gate
DRM-infested Books/376217648.epub
    Steven Erikson
    Reaper’s Gale
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/376227359.epub
    Jordan, Robert
    Winter’s Heart
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/376227401.epub
    Jordan, Robert
    Crossroads of Twilight
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/376227406.epub
    Jordan, Robert
    Knife of Dreams
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/376227409.epub
    Sanderson, Brandon
    The Gathering Storm
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/376227423.epub
    Jordan, Robert
    New Spring
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-free     Books/376231110.epub
    Cook, Glen
    Surrender to the Will of the Night
DRM-infested Books/376231528.epub
    Robert Jordan and Brandon Sanderson
    Towers of Midnight
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/378317076.epub
    Jordan, Robert
    A Crown of Swords
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/378317808.epub
    Robert Jordan
    Lord of Chaos
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-free     Books/379451459.epub
    Le Guin, Ursula K.
    Word for World is Forest, The
DRM-free     Books/37EC6E895E6BD70BEB48D9F1553D608E.epub
    Eben Hewitt
    Cassandra: The Definitive Guide
DRM-free     Books/380490608.epub
    Jordan, Robert
    The Eye of the World
DRM-free     Books/380494444.epub
    Asimov, Isaac
    The End of Eternity
DRM-infested Books/381497257.epub
    Harold McGee
    On Food and Cooking, The Science and Lore of the Kitchen
DRM-infested Books/381622032.epub
    IAIN M. BANKS
    Look to Windward
DRM-infested Books/381683084.epub
    Ursula K. Le Guin
    Tehanu
DRM-infested Books/381935940.epub
    Richard Adams
    Watership Down
DRM-infested Books/382674388.epub
    Steven Erikson
    Dust of Dreams
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/383912791.epub
    Steven Erikson
    Bauchelain and Korbal Broach
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/385975662.epub
    Jordan, Robert
    The Dragon Reborn
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/385981104.epub
    Steven Erikson
    The Bonehunters
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/385981116.epub
    Steven Erikson
    Midnight Tides
DRM-free     Books/385982966.epub
    Brust, Steven
    To Reign in Hell
DRM-infested Books/385987858.epub
    Steven Erikson
    Gardens of the Moon
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/385989170.epub
    Steven Erikson
    House of Chains
DRM-infested Books/385992628.epub
    Jordan, Robert
    The Fires of Heaven
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/385992927.epub
    Steven Erikson
    Toll the Hounds
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/385992930.epub
    Jordan, Robert
    The Great Hunt
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/385998417.epub
    Jordan, Robert
    The Shadow Rising
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/386016540.epub
    Esslemont, Ian C. C.
    Night of Knives
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/388403394.epub
    Steven Erikson
    The Crippled God
DRM-infested Books/389191300.epub
    Iain M. Banks
    Surface Detail
DRM-infested Books/390877859.epub
    Loewen, James W.
    Lies My Teacher Told Me
DRM-infested Books/393310992.epub
    Erikson, Steven
    Memories of Ice
DRM-infested Books/394745271.epub
    Steven Erikson
    Deadhouse Gates
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-free     Books/394745833.epub
    Walton, Jo
    Among Others
DRM-free     Books/395536306.epub
    Sir Arthur Conan Doyle
    The Adventures of Sherlock Holmes
DRM-free     Books/395537209.epub
    Sir Arthur Conan Doyle
    The Sign of the Four
DRM-free     Books/395539542.epub
    Sir Arthur Conan Doyle
    A Study in Scarlet
DRM-free     Books/395540660.epub
    Sir Arthur Conan Doyle
    The Hound of the Baskervilles
DRM-free     Books/395686685.epub
    Dante Alighieri
    Divine Comedy, Longfellow’s Translation, Complete
DRM-free     Books/395688318.epub
    Edgar Rice Burroughs
    A Princess of Mars
DRM-free     Books/395688375.epub
    Lafcadio Hearn
    Glimpses of an Unfamiliar Japan / First Series
DRM-infested Books/395926792.epub
    Fukuyama, Francis
    Origins of Political Order
DRM-infested Books/396269736.epub
    Herbert, Frank
    Chapterhouse: Dune
DRM-infested Books/398283114.epub
    Rothfuss, Patrick
    The Wise Man’s Fear
DRM-free     Books/3A5FBC58E821CFDF15C8C4E85657481E.epub
    Jon Hicks
    The Icon Handbook
DRM-free     Books/410943153.epub
    Edwin A. Abbott (A Square)
    Flatland: A Romance of Many Dimensions
DRM-free     Books/413463878.epub
    Brust, Steven
    Tiassa
DRM-free     Books/418293515.epub
    Heinlein, Robert A.
    Glory Road
DRM-infested Books/419950945.epub
    Isaac Asimov
    Foundation
DRM-infested Books/419950970.epub
    Isaac Asimov
    Foundation and Empire
DRM-infested Books/419950976.epub
    Isaac Asimov
    Second Foundation
DRM-infested Books/419968238.epub
    Scott Lynch
    The lies of Locke Lamora
DRM-infested Books/419968784.epub
    Scott Lynch
    Red Seas Under Red Skies
DRM-infested Books/420037362.epub
    Kim Stanley Robinson
    The Years of Rice and Salt
DRM-infested Books/420281728.epub
    Richard Wiseman
    59 Seconds: Think a Little, Change a Lot
DRM-infested Books/420445771.epub
    Isaac Asimov
    Foundation’s Edge
DRM-infested Books/420446058.epub
    Isaac Asimov
    Foundation and Earth
DRM-infested Books/420725428.epub
    Mike Resnick
    Kirinyaga: A Fable of Utopia
DRM-infested Books/421025353.epub
    William Dalrymple
    The Last Mughal
DRM-free     Books/421124117.epub
    Brust, Steven
    The Desecrator
DRM-infested Books/422530144.epub
    Max Barry
    Machine Man
DRM-free     Books/422718511.epub
    Apple Inc.
    Mac Integration Basics
DRM-free     Books/426914658.epub
    Brust, Steven
    Five Hundred Years After
DRM-free     Books/428235697.epub
    Vinge, Vernor
    A Fire Upon The Deep
DRM-infested Books/429173089.epub
    Ursula K. Le Guin
    The Other Wind
DRM-infested Books/429173713.epub
    Ursula K. Le Guin
    Tales from Earthsea
DRM-infested Books/429699133.epub
    Rajaniemi, Hannu
    The Quantum Thief
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/431617578.epub
    Walter Isaacson
    Steve Jobs
DRM-infested Books/432519291.epub
    Susan Weinschenk
    100 Things: Every Designer Needs to Know About People
DRM-infested Books/434509014.epub
    Stross, Charles
    Rule 34
DRM-free     Books/434522188.epub
    Larry Niven, Jerry Pournelle
    The Mote In God’s Eye
DRM-free     Books/434811509.epub
    Asher, Neal
    Cowl
DRM-infested Books/436646026.epub
    Neal Stephenson
    Reamde
DRM-infested Books/436691174.epub
    Julia Child
    Mastering the Art of French Cooking
DRM-infested Books/443149884.epub
    Daniel Kahneman
    Thinking, Fast and Slow
DRM-infested Books/446155927.epub
    Esslemont, Ian C. C.
    Stonewielder
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-free     Books/447591195.epub
    Asher, Neal
    The Skinner
DRM-infested Books/454252718.epub
    William B. Norton
    The Internet Peering Playbook: Connecting to the Core of the Internet
DRM-infested Books/455525627.epub
    Amar Chitra Katha
    Birbal The Genius
DRM-infested Books/458461362.epub
    Pamela Druckerman
    Bringing Up Bebe
DRM-free     Books/45B90418E467D479DCDDF23B932C648C.epub
    Douglas Crockford
    JavaScript: The Good Parts
DRM-infested Books/460822066.epub (directory)
    Scott Lynch
    The Republic of Thieves
DRM-free     Books/465679A557523FDB836005CF4BB9380E.epub
    Scott Berkun
    Mindfire
DRM-infested Books/479594044.epub
    Saladin Ahmed
    Throne of the Crescent Moon
DRM-infested Books/479717436.epub
    David Crist
    The Twilight War: The Secret History of America’s Thirty-Year Conflict with Iran
DRM-infested Books/479771801.epub
    William Dalrymple
    In Xanadu
DRM-infested Books/489957500.epub
    Bruce Schneier
    Liars and Outliers
DRM-infested Books/491186678.epub
    James Blish
    Cities in Flight
DRM-infested Books/491668459.epub
    Neal Asher
    Shadow of the Scorpion
DRM-infested Books/491669284.epub
    Glen Cook
    A Matter of Time
DRM-infested Books/491669288.epub
    Glen Cook
    Darkwar
DRM-free     Books/492199230.epub
    Frederik Pohl
    The Tunnel Under the World
DRM-free     Books/492199569.epub
    Frederik Pohl
    The Knights of Arthur
DRM-infested Books/494939678.epub
    Esslemont, Ian C. C.
    Orb Sceptre Throne
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/498634992.epub
    Charles Stross
    The Apocalypse Codex
DRM-infested Books/499392787.epub
    Daniel Suarez
    Kill Decision
DRM-free     Books/4AE7DDA54BEEFCD65157927546B18063.epub
    Roberto Ierusalimschy
    Programming in Lua 2ed
DRM-free     Books/4DCFF682728B765A1CE221F3D7C21536.epub
    Glen Cook
    Starfishers Volume 3: Stars’ End
DRM-free     Books/501278407.epub
    Colette
    Chéri
DRM-free     Books/501758197.epub
    David Brin
    Existence
DRM-infested Books/501758516.epub
    Scalzi, John
    Redshirts
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/503019669.epub
    J.R.R. Tolkien
    The Lord of the Rings
DRM-infested Books/503153300.epub
    J.R.R. Tolkien
    Tales from the Perilous Realm
DRM-infested Books/503154129.epub
    J. R. R. Tolkien and Christopher Tolkien
    The Book of Lost Tales, Part One
DRM-infested Books/503154991.epub
    J. R. R. Tolkien and Christopher Tolkien
    The Book of Lost Tales, Part Two
DRM-infested Books/503155327.epub
    J.R.R. Tolkien
    The Children of Húrin
DRM-infested Books/503163148.epub
    J.R.R. Tolkien
    The Hobbit Deluxe
DRM-infested Books/503164678.epub
    J.R.R. Tolkien
    The Silmarillion
DRM-infested Books/503167303.epub
    J.R.R. Tolkien
    Unfinished Tales of Númenor and Middle-earth
DRM-infested Books/504209078.epub
    Isaac Asimov
    Prelude to Foundation
DRM-infested Books/504371982.epub
    Iain M. Banks
    The Hydrogen Sonata
DRM-free     Books/511060740.epub
    Frederik Pohl
    The Hated
DRM-free     Books/511143617.epub
    Frederik Pohl
    The Day of the Boomer Dukes
DRM-free     Books/511252896.epub
    Frederik Pohl
    Pythias
DRM-free     Books/513357605.epub
    Hannu Rajaniemi
    The Fractal Prince
DRM-free     Books/513868CF0AA46D293EE27F74BC399760.epub
    Jerry Pournelle
    West of Honor
DRM-infested Books/520233773.epub
    Nate Silver
    The Signal and the Noise
DRM-free     Books/520897403.epub
    Hugh Howey
    Wool Omnibus
DRM-free     Books/525170910.epub
    Cory Doctorow and Charles Stross
    The Rapture of the Nerds
DRM-infested Books/526136048.epub (directory)
    Hetty van de Rijt & Frans Plooij
    The Wonder Weeks
DRM-infested Books/529020127.epub
    Neal Asher
    The Departure
DRM-infested Books/529424632.epub
    Guy Gavriel Kay
    The Lions of Al-Rassan
DRM-infested Books/536312376.epub
    Glen Cook
    Garrett for Hire
DRM-free     Books/537023027.epub
    Erikson, Steven
    Forge of Darkness
DRM-infested Books/541673159.epub
    Ursula K. Le Guin
    The Tombs of Atuan
DRM-infested Books/541673162.epub
    Ursula K. Le Guin
    The Farthest Shore
DRM-infested Books/546126326.epub
    Jan Morris
    HAV
DRM-infested Books/551241606.epub
    Ursula K. Le Guin
    A Wizard of Earthsea
DRM-infested Books/551567785.epub
    Murray R. Spiegel, PhD
    Schaum’s Outline Mathematical Handbook of Formulas and Tables, Fourth Edition
DRM-infested Books/551747038.epub
    Chris Hedges and Joe Sacco
    Days of Destruction Days of Revolt v2b
DRM-infested Books/552144691.epub
    Tamim Ansary
    Games without Rules
DRM-free     Books/553878102.epub
    Charles Stross
    A Tall Tail
DRM-infested Books/563408849.epub
    Iain Banks
    Stonemouth
DRM-infested Books/568731449.epub
    William Dalrymple
    Return of a King: The Battle for Afghanistan, 1839-42
DRM-infested Books/569232538.epub
    Neil Gaiman
    The Ocean at the End of the Lane
DRM-free     Books/571678152.epub
    Glen Cook
    The Return of the Black Company
DRM-free     Books/571678945.epub
    Glen Cook
    The Many Deaths of the Black Company
DRM-free     Books/573656304.epub
    Glen Cook
    Chronicles of the Black Company
DRM-free     Books/573656441.epub
    Glen Cook
    The Books of the South
DRM-free     Books/576233114.epub
    Jack Vance
    Demon Princes
DRM-infested Books/578851675.epub
    Zilpha Keatley Snyder
    Below The Root
DRM-infested Books/580642602.epub
    Max Barry
    Lexicon
DRM-infested Books/588794444.epub
    Charles Stross
    Neptune’s Brood
DRM-free     Books/5CCB889F91585202637A3C5FBFD8409F.epub
    Glen Cook
    Starfishers-The Starfishers Trilogy Volume II
DRM-infested Books/600938002.epub
    Gardner Dozois
    The Year’s Best Science Fiction: Thirtieth Annual Collection
DRM-free     Books/606232503.epub
    Ian C. Esslemont
    Blood and Bone
DRM-free     Books/610920977.epub
    Robert Jordan and Brandon Sanderson
    A Memory of Light
DRM-free     Books/6122FF30560866BD75257E6CCC264371.epub
    Fritz Leiber
    Swords and Ice Magic-Fafhrd and the Gray Mouser-Book 6
DRM-infested Books/619483561.epub
    Raymond E. Feist
    Magician’s End
DRM-free     Books/622837311.epub
    Le Comte De  Lautréamont
    Les chants de Maldoror
DRM-free     Books/62497334A4189B1D00E9BDFD95724E2E.epub
    Fritz Leiber
    The Knight and Knave of Swords-Fafhrd and the Gray Mouser-Book 7
DRM-infested Books/645571245.epub
    Iain Banks
    The Quarry
DRM-free     Books/647688922.epub
    Steven Brust and Skyler White
    The Incrementalists
DRM-infested Books/651331715.epub
    Susan Crawford
    Captive Audience
DRM-infested Books/654456347.epub
    Iain Banks
    The Wasp Factory
DRM-infested Books/662310218.epub (directory)
    Various Authors
    Star Wars: Empire Volume 3 – The Imperial Perspective
DRM-infested Books/662310219.epub
    Paul Gulacy
    Star Wars: Crimson Empire
DRM-infested Books/664297397.epub (directory)
    Various Authors
    Star Wars: Empire, Vol. 4: The Heart of the Rebellion
DRM-infested Books/664343525.epub (directory)
    Paul Chadwick, Doug Wheatley & Tomás Giorello
    Star Wars: Empire, Vol. 2: Darklighter
DRM-infested Books/664894567.epub
    John Ostrander
    Star Wars: Dawn of the Jedi Volume 1—Force Storm
DRM-infested Books/664910993.epub (directory)
    Scott Allie, Ryan Benjamin & Brian Horton
    Star Wars: Empire Vol. 1
DRM-free     Books/666772E4C21E2017E71C10F1990840BC.epub
    Pieter Hintjens
    ZeroMQ - Connecting your Code
DRM-infested Books/674224604.epub (directory)
    Various Authors
    Star Wars: Empire, Vol. 5: Allies and Adversaries
DRM-infested Books/674226603.epub (directory)
    Thomas Andrews, Scott Allie & Various Authors
    Star Wars: Empire, Vol. 6: In the Shadows of Their Fathers
DRM-free     Books/697938901.epub
    Charles Stross
    Equoid: A Laundry Novella
DRM-free     Books/6BBD61A64A46FFFF4AE7C5FCEB9CFCEE.epub
    David Drake
    Balefires
DRM-free     Books/71C79A253586282ABE73C2975237EB08.epub
    Glen Cook
    Sung in Blood
DRM-free     Books/73BC5DD0E51611BDC359CBAB48CC203F.epub (directory)
    Crane, Stephen
    The Red Badge of Courage
DRM-free     Books/7A0D1AEE343638A8A3769DABB90CD4CB.epub
    Clay A. Johnson
    The Information Diet
DRM-free     Books/7B46914B55481714DED3D711288978FA.epub
    Glen Cook
    Shadowline-The Starfishers Trilogy I
DRM-free     Books/809DCC75FE60E749458CB27636EE6777.epub
    Mercedes Lackey
    The Secret World Chronicle
DRM-free     Books/8350F5DFECCD6AA88714400FDF4F6831.epub
    François de La Rochefoucauld
    Réflexions ou Sentences et Maximes Morales
DRM-free     Books/853BD1C65277D276BA09E04EBFEB73EF.epub
    None
    PostgreSQL 9 Administration Cookbook
DRM-free     Books/85907063B2351E91C6E7F5052C090BFD.epub
    None
    PostgreSQL 9.0 High Performance
DRM-free     Books/8663E7BF735C06318FF450532A67F1C2.epub
    Glen Cook
    Passage at Arms
DRM-free     Books/8690D69995483C5D2DF6AD38BE53C1D7.epub
    Paolo Bacigalupi
    The Windup Girl - Second Electronic Edition
DRM-free     Books/8993920EF1C0E8FE8CB47A20BD955F53.epub
    Fritz Leiber
    Swords Against Death-Fafhrd and Gray Mouser-Book 2
DRM-free     Books/8DC6ED0A99161EBB516E370A60ED1121.epub
    Jonathan Zdziarski
    Hacking and Securing iOS Applications
DRM-free     Books/8DD0D22CD05B0B8D52DF9DB93FC8616B.epub
    Fritz Leiber
    Swords in the Mist-Fafhrd And the Gray Mouser-Book 3
DRM-free     Books/912CB8B110736684549EAC4FC36665AB.epub
    Neil Gaiman and Dave McKean
    Signal to Noise
DRM-free     Books/93061FDFD6EEC01FD8CE4295A049C97C.epub
    Cory Doctorow
    Homeland
DRM-free     Books/9592D5001632B94D1FEFC98B3A40E049.epub
    Kelly Link
    Stranger Things Happen
DRM-free     Books/990CA5799084407151488A7C563DF269.epub
    Tom Hughes-Croucher
    Node: Up and Running
DRM-free     Books/9CCCA9E217602948B84BE9A7A21C2753.epub
    Mike Resnick
    Birthright: The Book of Man
DRM-free     Books/9DA2C8D7E941C37FA83192E5849AA850.epub
    Q. Ethan McCallum
    Parallel R
DRM-free     Books/A7DB557515983DB67F74E2BE351FC319.epub
    Glen Cook
    Reap the East Wind
DRM-free     Books/B18AB85267B815E540C7E370F8D97726.epub
    Lars George
    HBase: The Definitive Guide
DRM-free     Books/B1A586D24AE5F2450010F8664F8E059D.epub
    Cory Doctorow
    Pirate Cinema
DRM-free     Books/B562B5D74F8677C946B0A6F81EB34F1B.epub
    Gotthold Ephraim Lessing
    Nathan the Wise; a dramatic poem in five acts
DRM-free     Books/BE42D426242836AA171539B7415732E6.epub
    Glen Cook
    The Swordbearer
DRM-free     Books/BE80AD93781746E45CF95607A7BEE687.epub
    Charlie Stross
    Bit Rot
DRM-free     Books/C39FBC59673713A57D404D62BE85C4DC.epub
    Lauren Beukes
    Zoo City
DRM-free     Books/C81CBBD470EFDBC62359BDAD12FDF551.epub
    Thomas Hobbes
    Leviathan
DRM-free     Books/D17DC1A83BD90AFB24055A01831BFAB4.epub
    Glen Cook
    The Dragon Never Sleeps
DRM-free     Books/DA8FBAB386EC503A0EF21E481D214521.epub
    David Flanagan
    JavaScript: The Definitive Guide
DRM-free     Books/DBAD0DE7A4305B90F6AC33C673FE68E8.epub
    Glen Cook
    A Path to Coldness of Heart
DRM-free     Books/DBCCB54B140D1158250B24B7E3E81B63.epub
    Scott Chacon
    Pro Git
DRM-free     Books/DF6C56F9A8719294B13BA91BED5E1667.epub
    Mike Resnick
    Ivory
DRM-free     Books/E8D76FDD351E68D92AE4A3F3AEA7EC6A.epub
    Paolo Bacigalupi
    Pump Six and Other Stories
DRM-free     Books/EEE65A68378C2510E18DF24CD767AC9A.epub
    Fritz Leiber
    Swords Against Wizardry-Fafhrd and the Gray Mouser-Book 4
DRM-free     Books/F3BC14466A16C96CCD8FFE00DCCF8147.epub
    Glen Cook
    An Empire Unacquainted With Defeat
DRM-free     Books/F534835595041374B814D151A633E69D.epub
    Peter Watts
    Blindsight
DRM-free     Books/F8BF760284ADC800B290DCA6D8EA7EF2.epub
    Ben Klemens
    21st Century C
DRM-free     Books/FE8C57863E40B98CB732FEE4BFDB60BB.epub
    Glen Cook
    An Ill Fate Marshalling
8 books are DRM-infested
26 could be cured

Update (2013-11-06):

OS X 10.9 Mavericks and the new iBooks app changed the location of the iBooks directory, I changed my script accordingly (and made it adjust depending on which OS version you have). Also, the file names have changed and no longer embed author and title, so I am extracting them from the XML metadata files.

The Gresham’s law of Amazon Web Services

In the bad (good?) old days when currency’s worth was established by the amount of gold or silver in coinage, kings would cut corners by debasing currency with lead, which is almost as dense as gold or silver. In the New World, counterfeiters debased gold coins with platinum, which was first smelted by pre-columbian civilizations. Needless to say, the fakes are now worth more than the originals.

The public was not fooled, however, and found ways to test coins for purity, including folkloric ones like biting a coin to see if it is made of malleable gold, rather than harder metals. People would then hoard pure gold coins, and try to rid themselves of debased coins at the earliest opportunity. This led to Gresham’s Law: bad money drives out good money in circulation.

After a year of using Amazon Web Services’ EC2 service at scale for my company (we moved to our own servers at the end of 2011), I conjecture there is a Gresham’s Law of Amazon EC2 instances – bad instances drive out good ones. Let me elaborate:

Amazon EC2 is a good way to launch a service for a startup, without incurring heavy capital expenditures when getting started and prior to securing funding. Unfortunately, EC2 is not a quality service. Instances are unreliable (we used over 80 instances at Amazon, and there was at least one instance failure a week, and sometimes up to 4). Amazon instances have poor disk I/O performance that makes them particularly unsuitable to hosting non-trivial databases (EBS is even worse, and notoriously unreliable).

Performance is also inconsistent—I routinely observed “runt” m1.large instances that performed half as well as the others. We experienced all sorts of failure modes, including disk corruptions, disks that would block forever without timing out, sporadic losses of network connectivity, and many more. Even more puzzling, I would get 50% to 70% failure rate on new instances that would not come up cleanly after being launched.

Some of this is probably due to the fact we use an uncommon OS, OpenSolaris, that is barely supported on EC2, but I suspect a big part of this is that Amazon uses low-end commodity parts, and does not proactively retire failed or flaky hardware from service. Instances that have the bad luck of being assigned to flaky hardware are more likely to fail or perform poorly, and thus more likely to be be destroyed, released and a new one reassigned in the same slot. The inevitable consequence of this is that new instances have a higher likelihood of being runts or otherwise defective than long-running ones.

One work-around is to spin up a large number of instances, test them, and destroy the poor-performing ones. AWS runts are usually correlated with slower CPU clock speeds, as older machines would be running older versions of the Xen hypervisor Amazon uses under the hood, have less cache, slower drives and so on. Iterating through virtual machines as if you are picking melons at a supermarket is a slow and painful job, however, and even their newer machines have their share of runts. We were trying to keep only machines with 2.6 or 2.66GHz processors, but more than 70% of the instances we were getting assigned were 2.2GHz runts, and it would usually take creating 5 or 6 instances on average to get a non-runt.

In the end, we migrated to our own facility in colo, because Amazon’s costs, reliability and performance were just not acceptable, and we had long passed the threshold beyond which it is cheaper to own than rent (I estimate it at $5,000 to $10,000 per month Amazon spend, depending on your workload). It is not as if other cloud providers are any better—before Amazon we had started on Joyent, which supports OpenSolaris natively, and their MTBF was in the order of 2 weeks, apparently because they replaced their original Sun hardware with substandard Dell servers and had issues with power management C-states in the Dell server BIOS.

The dirty secret of cloud services is that there is no reliable source of information on actual performance and reliability of cloud services. This brings out another economic concept, George Akerlof’s famous paper on the market for lemons. In a market where information asymmetry exists, the market will eventually collapse in the absence of guarantees. Until Amazon and others offer SLAs with teeth, you should remain skeptical about their ability to deliver on their promises.

Withings smart baby monitor review

One of the joys challenges of being a first-time parent is being exposed to a bewildering array of gadgets and equipment required to care for the baby, from baby car seats, strollers and diaper pails to 2-axis rocking robots (thanks Rohit!). There is an entire cottage industry of books like Baby Bargains that help you navigate through the confusing and sometimes questionable or outright unnecessary choices.

I have a Withings body weight scale that I really like and I was excited to learn they were going to release a networked video baby monitor. It took a while to get to market in the US, however, so in the interim I purchased a Philips Advent DECT digital baby monitor, which ended up unusable in practice, because its microphone sensitivity is so poor that you can barely hear anything. When the Withings baby monitor finally became available in the US, I immediately ordered it.

Withings is clearly taking design cues from Apple, from the lavishly designed packaging to the glossy white plastic RoundedRect aesthetic and the use of a magnetic clip to attach the baby monitor to the crib. The clip is serviceable, but the magnets are not quite strong enough to hold the unit firmly onto the crib. I would not trust it to keep the monitor from toppling when the baby grows and kicks at the crib. Fortunately they also include a flip-out tab on the base of the unit that can be inserted into a slit on the clip to prevent sliding, although it is not obvious and it took me a while before I discovered this key feature.

The wall wart is a generic black model with swappable AC prongs for international markets, and detracts from the overall package, but since the monitor has a micro-USB input, you can always use another standard AC to USB type A adapter like the iPhone’s, with a USB type A to micro-USB cable. A rechargeable battery is included, with 2 hours’ claimed life, I did not verify that spec.

The initial out of the box experience is good: you connect to the device from your iPhone or iPad using Bluetooth (no messing around with a USB cable as with the Withings scale), enter the WiFi settings in the Withbaby app, and then use WiFi to access the device afterwards. It is as streamlined an experience as you can expect without a keyboard on the unit. There is also an Ethernet jack (it is unclear whether it supports power over Ethernet), but my house was built in 1928 and is not wired upstairs where the baby lives.

Once you enter your credentials into the app, it connects to the monitor and shows you the video and sound. If you put it in the background, you have the option of monitoring audio. Withings will also send you alerts via push notifications if the temperature or humidity is excessive, or if it detects noise or motion. The default settings are way too twitchy, however, and you will find yourself disabling audio notifications as the deluge of alerts is just too much.

The device includes a night light with selectable color, a lullaby player, and the ability to speak to your baby, all controlled through the app. At the front you also have touch controls to turn some of these features on. This is actually a bad idea, as on two occasions I started the lullaby by accident as I was fumbling with it in a dark room, and woke up my baby as a result. Another design flaw is the pulsing blue night light when the unit is rebooting, the Airport Express like amber/green status LED in the back is quite sufficient. Frankly the only one of these features that is useful is the speaker, and the ability to stream from your music collection, such as Dr. Harvey Karp’s white noise selections would be preferable to the canned lullabies.

The video camera is advertised as having a 3 megapixel sensor. It has a wide-angle lens and you can “pan” using the usual iPhone or iPad gestures. The lens is a fixed-focus plastic one, and optical clarity is so-so at best, optimal focus seems to be at 50cm or so. One great feature is the monitor has a normal and night vision mode, similar to the one on some Sony HAD camcorders, with an IR illuminator that provides light for the night vision mode. This means you can watch your baby toss and turn in an otherwise pitch-black room.

You can use the baby monitor from outside your network, and it works fine, even over a 3G connection. Withings allows you up to 15 minutes per day, anything beyond that requires paying them $6 for each 100 minutes. Coming on top of an already expensive device, this seems like a naked money grab from anxious parents. (Updated 2012-09-29: remote monitoring is now free and unlimited).

When the unit works, it is absolutely great: good sound sensitivity and the video feature mostly works as advertised. Unfortunately it frequently does not function, and I find myself performing a hard reboot by removing the battery far more often than I would like. Among the pathologies:

  • Once it falsely reported the unit was closed and thus video inaccessible
  • Once the camera was in a frozen state, it took a power cycling to get the video moving again.
  • Yesterday I could not connect at all, no matter how many times I rebooted my Airport Extreme, the monitor and my wife’s or my iPads. Some detective work using a packet sniffer showed the app was trying to connect to babyws.withings.net using HTTP, which is aliased to s11.withings.net, and that server was down. Some of the documentation suggests you can use the Bluetooth connection to access the monitor, but I was not able to figure out how to do this.

This brings me to a crucial point. The baby monitor is a safety device, and it is utterly unacceptable for its functioning to be dependent on a cloud service, which can and will be a single point of failure. It should use Bonjour or similar discovery methods to work on the LAN, and rely on Withings’ servers only when accessing it from outside the home LAN’s perimeter. I wonder if Withings’ eagerness to nickel-and-dime users by charging for outside monitoring led to this critical design flaw.

The bottom line is the Withings smart baby monitor is a very frustrating device, with its obvious potential marred by failures of execution. If it worked consistently, it would be a top-notch product worthy of its Apple inspiration and lofty price tag, but the general lack of reliability means I cannot recommend it until the bugs are ironed out. Consider it an alpha release at best.

Update (2012-09-17):

Here’s how to make the Withings not-so-smart baby monitor more usable:

  • Remove the battery from the unit and hook up the micro-USB power adapter to a Belkin WeMo remote-controlled power switch. This allows you to power-cycle the baby monitor remotely from the same iPhone or iPad you are using the monitor software on.
  • Hide the blue led with gaffer’s tape. This prevents the blue light on reboot from waking the baby. Unlike duct tape, gaffer’s tape can be removed without leaving glue residue, although the aesthetics of dark gray gaffer’s tape on the gleaming white unit are questionable at best.
  • I haven’t tried covering up the touch controls with gaffer’s tape, which would eliminate the risk of triggering a jingle and waking the baby. The WeMo eliminates the need to enter the room and tinker with the baby monitor.

It’s quite sad to have to pay an extra $50 to work around buggy hardware and software, but it makes a big difference.

Update (2013-05-20):

The micro-USB connector failed and the baby monitor is now essentially a doorstop. Not surprising given how flimsy micro-USB is, compared to mini-USB, for insignificant space savings. Micro-USB was a Nokia design rammed through the USB-IF. In theory it has better insert-remove cycle life than mini-USB, but in practice I’ve never had mini-USB fail, whereas it is a frequent occurrence with micro-USB.

Update (2015-08-17):

USB-C is an improvement over micro-USB, hopefully some future version of the baby monitor will use it. Still nowhere near as robust as Lightning or tip-ring-sleeve, though.

The HP-15C was reissued at long last!

It is strange no one seems to have picked up the news yet, but HP has reissued the legendary HP-15C in a special “30th anniversary limited edition”, and it became available for purchase last week.

HP-15C Limited Edition

The new HP-15C is not strictly speaking a reissue but a replica, as it does not use the original’s Saturn processor, but instead an emulation thereof running on an ARM CPU. Even emulated, it should be much faster than the original 640 _kilohertz_ processor. I ordered two, and received them today.

As expected, the quality is in line with the current HP-12C, i.e. not as good as the 1980s models in terms of key feel, but still leagues ahead of any competing product. The originals used a special 47-point bonding process to ensure the utmost in rigidity and reliability, I doubt the current model had as much attention paid to detail. It is made in China, obviously, the Corvallis facility is long gone. The slipcover fits very poorly (too tight, and the seams are not trimmed properly) and feels thinner and outright cheap compared to the original. The labels on the keys are accurately positioned, at least, unlike the train wreck that was the HP-12C Platinum. The cheat sheet in the back is a garish black on silver as on the 35S, instead of the original’s silver on black. It also uses two 3V CR2032 batteries instead of the 3 button cells in  the original.

Speed-wise, the Limited Edition integrates the normal distribution nearly instantly, when that test that took 34 seconds on the original.

In short: not as good as the original, but still an excellent calculator for those who prize ergonomics.

A waiter for a server

I had to monitor a long-running process on a Solaris server tonight, but didn’t want to stay glued at a computer monitor. A neat trick:

ssh myserver.example.com "pwait 17601"; say "batch done"

You would replace 17601 with the process ID of the job you are waiting for, of course. That way, my Mac connects to the server, waits for the job to complete, then gives me an spoken alert when it is done. I can watch a movie, do chores or whatever during that time. I am sure there are equivalent commands to pwait for Linux.

Will Adobe ever learn?

In a triumph of hope over experience, I recently “upgraded” from Adobe CS3 Design Standard to CS5 Design Standard. I hardly ever use Photoshop any more since I started using Aperture and Lightroom (originally a Macromedia product, no matter what the lame “Adobe Photoshop Lightroom” face-saving branding may try to claim), the main driver for the purchase was actually InDesign CS5 and its ePub functions.

Of course, this is Adobe. Previous versions gratuitously included crud like a full Opera install (an older version, insecure, naturally) just to display a splash screen, or a full MySQL install to power Acrobat search. I never install Acrobat, of course, since that bloated and bug-ridden piece of garbage managed to steal the crown for most insecure software from Internet Explorer, no small feat.

Adobe does not want to confuse users with streamlined and efficient software, so they decided to include the mostly useless Growl on-screen notification program to nag you into registering. Increasing bloat and attack surface for malware is not a good idea, nor is interrupting creative people’s flow with interruptions. Of course, helping clients Get Things Done is a low priority at Adobe, as evidenced by their product choices.

You have to pity the Growl developers, whose reputation will suffer from guilt by association. I dislike interruptions and do not find it useful, but many people do and rave about it. They installed it by choice, not as a sneaky drive-by install for slimy marketing purposes.

Some more annoyances in CS5:

  • The pricing for the suite is more than the sum of its parts: $200 each for Photoshop, InDesign or Illustrator, $700 for Design Standard. I suppose they must think Acrobat and their online tie-ins have a value of $100 (hint: they forgot the negative sign).
  • Of course, they won’t let you upgrade individual component applications.
  • On the plus side, they now have the decency to include Acrobat on a separate CD, so you can discard it immediately and not risk installing it as a side-effect of installing the apps that are actually useful.
  • The icons were designed by the world’s laziest and most creatively bankrupt designer, just as with CS3 and CS4
  • Performance on a high-end 8-core or 12-core Mac is actually slower than on a lower-end configuration, thanks to legacy cruft and incompetence.
  • It is slower to load on my wife’s MacBook Pro. Each successive version of OS X is faster on the same hardware, Microsoft and Adobe deliver software that gets progressively slower.

In other words, unlike Lightroom, CS5 is designed to be endured, not to delight.

Incensed at Mozilla

One of the greatest features in the Webkit-based browsers (Apple’s Safari and Google Chrome) is WebSQLdatabase, the ability for a web site to store information in a SQLite database on your browser accessible via JavaScript. This allows web developers to build database-enabled applications that run entirely in the browser, without requiring a server. This is very useful for mobile devices, which in the US enjoy flaky network connectivity at best. One very handsome example is the iPad-optimized Every Time Zone webapp.

SQLite is probably the most important open-source project you have never heard of. It is a simple, streamlined and efficient embedded database. Firefox stores its bookmarks in it. Google distributes its database of phishing sites in that format. Sun’s industrial-strength Solaris operating system stores the list of services it runs on boot in it—if it were to fail, a server would be crippled so that is a pretty strong vote of confidence. Adobe Lightroom and Apple’s Aperture use it to store their database, as do most Mac applications that use the CoreData framework, and many iPhone apps. In other words, it is robust and proven mission-critical software that is widely yet invisibly deployed.

WebSQLdatabase basically makes the power of SQLite available to web developers trying to build apps that work offline, specially on mobile devices. No good deed goes unpunished, and the Mozilla foundation teamed up with unlikely bedfellow Microsoft to torpedo formal adoption of WebSQLdatabase as a web standard, on spurious grounds, and pushed an alternate standard called IndexedDB instead. To quote the Chromium team:

Q: Why this over WebSQLDatabase?

A: Microsoft and Mozilla have made it very clear they will not implement SQL in the browser.  If you want to argue this is silly, talk to them, not me.

IndexedDB is several steps backwards. Instead of using powerful, expressive and mature SQL technology, it uses a verbose JavaScript B-tree API that is a throwback to the 1960s bad old days of hierarchical databases and ISAM, requires a lot more work from the developer, for no good reason. To add injury to insult, Firefox 4’s implementation of IndexedDB is actually built on top of SQLite. The end result will be that web developers will need to build a SQL emulation library on top of IndexedDB to restore the SQLite functionality deliberately crippled by IndexedDB. If there is one constant in software engineering, it is that multiple layers add brittleness and impair performance.

Of course, both Mozilla and Microsoft are irrelevant on mobiles, where WebKit has essentially won the day, so why should this matter? Microsoft has always been a hindrance to the development of the web, since they have to protect the Windows API from competition by increasingly capable webapps, but I cannot understand Mozilla’s attitude, except possibly knee-jerk not-invented-here syndrome and petulance at being upstaged by WebKit. WebSQLdatabase is not perfect—to reach its full potential, it needs and automatic replication and sync facility between the local database and the website’s own database, but it is light years ahead of IndexedDB in terms of power and productivity.

I am so irritated by Mozilla’s attitude that after 10 years of using Mozilla-based browsers, I switched today from Firefox to Chrome as my primary browser. Migrating was surprisingly easy. Key functionality like bookmark keywords, AdBlock, FlashBlock, a developer console, and the ability to whitelist domains for cookies, all have equivalents on Chrome. The main regressions are bookmark tags, and Chrome’s sync options are not yet equivalent to Weave‘s. At some point I will need to roll my own password syncing facility (Chrome stores its passwords in the OS X keychain, which is also used by Safari and Camino).

RapidSSL 1 – GoDaddy 0

My new company’s website uses SSL. I ordered an “extended validation” certificate from GoDaddy, instead of my usual CA, RapidSSL/GeoTrust, because GoDaddy’s EV certificates were cheap. EV certificates are security theater more than anything else, I probably should not have bothered.

Immediately after switching from my earlier “snake oil” self-signed test certificate to the production certificate, I saw SSL errors on Google Chrome for Mac and Safari for Mac, i.e. the two browsers that use OS X’s built-in crypto and certificate store. I suppose I should have tested the certificate on another server before going live, but I trusted GoDaddy (they are my DNS registrars, and competent, if garish).

Big mistake.

I called their tech support hotline, which is incredibly grating because of the verbose phone tree that keeps trying to push add-ons (I guess it is consistent with the monstrosity that is their home page).

After a while, I got a first-level tech. He asked whether I saw the certificate error on Google Chrome for Windows. At that point, I was irate enough to use a four-letter word. Our customers are Android mobile app developers. A significant chunk of them use Macs, and almost none (less than 5%) use IE, so know-nothing “All the world is IE” demographics are not exactly applicable.

After about half an hour of getting the run-around and escalating to level 2, with my business partner Michael getting progressively more anxious in the background, the level 1 CSR tells me the level 2 one can’t reproduce the problem (I reproduced it on three different Macs in two different locations). I gave them an ultimatum: fix it within 10 minutes or I would switch. At this point, the L1 CSR told me he had exhausted all his options, but I could call their “RA” department, and offered to switch me. Inevitably, the call transfer failed.

I dialed their SSL number, and in parallel started the certificate application process on RapidSSL. They offered a free competitive upgrade, I tried it, and within 3 minutes I had my fresh new, and functional certificate, valid for 3 years, all for free and in less time than it takes to listen to GoDaddy’s obnoxious phone tree (all about “we pride ourselves in customer service” and other Orwellian corporate babble).

I then called GoDaddy’s billing department to get a refund. Surprisingly, the process was very fast and smooth. I guess it is well-trod.

The moral of the story: GoDaddy—bad. RapidSSL—good.

Update (2012-08-26)

I switched my DNS business from GoDaddy to Gandi.net in December 2011 after Bob Parsons’ despicable elephant-hunting stunt.

One month with the iPad

Since I got my iPad six weeks ago, I have only used my MacBook Air once.

I am not going to repeat the extensive reviews posted elsewhere, but after over a month of extensive use, give some perspective for those who don’t get the point of the iPad, or other similar devices.

First of all, commentators have focused on entirely the wrong thing: feeds and speeds, missing features like multitasking or Flash, Apple’s iron fist over app developers. The iPad begins and ends with the user experience, and that means multi-touch and the incredibly long battery life. That’s why comparisons to stylus-driven devices like the unsuccessful Microsoft Tablet PC miss the point. The amazing battery life, specially on standby (I have never managed to go under 60%, even after three days without charging), means you can use it as a real mobile device and not subconsciously watch the battery meter.

Is it a perfect device? Of course not. Mobile Safari has a hard time with complex and heavy pages like those from my Temboz RSS/Atom feed reader, the screen is too prone to reflections and fingerprints, and Apple’s use of high-quality materials like aluminium and glass instead of plastic and acrylic makes it heavier to hold than necessary.

As to whether it is a replacement for a laptop, the answer is yes and no. The iPad is the first in an entirely new class of devices, and I think it has the potential to replace desktop and laptop computers as the dominant form of consumer computing. The touch user interface makes for a very engaging user experience, far more than using a mouse and keyboard ever did. To be sure, the input limitations do not make it a very efficient content creation device, but that’s where opinions diverge.

I use desktop computers for real work (an eight-core Mac Pro with 12G of RAM and a 30″ display at home, a quad-core iMac with a 27″ display at work). A laptop just feels too constricting for extended use. I have the luxury of using proper desktops because I do not travel much for work, and the extent of my mobile use is reading books or browsing the web while commuting by bus. The improvements that most benefit me are in synchronizing my iPad with multiple computers, and offline capability (I got the WiFi model since there is no way I will pay AT&T for their garbage excuse of a network).

Road warriors need a more featured device, even if cramped, and will not be so impressed. I think genuine mobile users are a minority, however. Surveys in the past showed that most laptops are tethered, i.e. users would unplug them from home, take them to work and plug them there, and back. That is why Windows laptop makers introduced monstrosities like Pentium 4 powered laptops with battery lives that barely exceeded the hour. Laptop sales exceeded those of desktops because many people wanted the option of mobility, even if they seldom, if ever, availed themselves of it, and a less obtrusive presence in their homes than the typical beige box with its rat’s warren of cables. Those people would be better served by a well-designed desktop like the iMac and an iPad for the occasional mobile use.

Clueless SaaS providers can leave you with egg on your face

While cleaning out my spam folders, I noticed a disturbing trend: a number of the spam were sent to vendor-specific email addresses I had set up to communicate with Parallels, Joyent and Shoeboxed. As a security measure, I do not give my personal email address to vendors, only aliases. The email address I used in the past for Dell was dell@majid.fm, for instance (I now use a different domain). A few years back, I started receiving pornographic spam at that address, which led me to think either Dell had secretly adopted a radically new diversification plan, or that their customer database had been compromised. Needless to say, this did not reflect well on Dell. I canceled that alias and stopped dealing with Dell.

I contacted the support for the three vendors. Joyent got back to me, and said:

We have traced this back to a third-party provider that was used to distribute service notifications. We have been in contact with this service provider, and they have determined that subscriber email addresses of their clients were compromised. They have launched their own investigation, which is ongoing, and have also reached out to their local FBI office.

After some digging, I found some interesting posts. Some email marketing company called iContact, that I had never heard about before, was the source of the compromise. They claim to be SAS-70 compliant, but of course like most bureaucratic certifications, SAS-70 is mostly security theater that makes sysadmins’ life miserable for no meaningful security benefit (SAS-70 auditors, on the other hand, profit handsomely).

Just another example of how outsourcing critical functions to outside vendors can backfire spectacularly and take down your own reputation in the process.

Just enough Weave

Note: I am keeping this code around for historical purposes, but it has not worked since Weave 1.0 RC2. I created this because Mozilla’s public sync servers were initially quite unreliable, but they have remedied the situation and performance problems are a thing of the past. I also learned the inner workings of Weave/Firefox Sync in the process, and am satisfied as to the security of the system. Since I no longer use Firefox myself, I do not expect to ever revive this project. Feel free to take it over, otherwise you are best served by using Mozilla’s cloud.

Like most of my readers, I use multiple computers: my Mac Pro at home, my MacBook Air when on the road, 3 desktop PCs at work, a number of virtual machines, and so on. I have Firefox installed on all of them. The Mozilla Weave extension allows me to sync bookmarks, passwords et al between them. Weave encrypts this data before uploading it to the server, but I do not like to rely on third-party web services for mission-critical functions (my Mozilla server was down last Monday, for instance, due to the surge of traffic from people returning to work and performing a full sync against 0.5). Through Weave 0.5, I ran my own instance of the Mozilla public Weave server version 0.3. Unfortunately, Weave 0.6 requires server version 0.5 and I had to upgrade.

The open-source Weave server is implemented in PHP. It doesn’t require Apache compiled with mod_dav as early versions did (I prefer to run nginx), but it is still a fairly gnarly piece of code that is anything but plug-and-play. Somehow I had managed to get version 0.3 running on my home server, but no amount of blundering around got me to a usable state with 0.5. I ended up deciding to implement a minimalist Weave server in Python, as it seemed less painful than continuing to struggle with the Mozilla spaghetti code, which confusingly features multiple pieces of code that appear to do exactly the same thing in three different places. Famous last words…

Three days of hacking later, I managed to get it working. 200 or so lines of Python code replaced approximately 12,000 lines of PHP. Of course, I am not trying to reproduce an entire public cloud infrastructure like Mozilla’s, just enough for my own needs, using the “simplest thing that works” principle. Interestingly, the Mozilla code includes a vestigial Python reference implementation of a Weave server for testing purposes. It does not seem to have been working for a while, though. I used it as a starting point but ended up rewriting almost everything. Here are the simplifying hypotheses:

  • My weave server is meant for a single user (my wife prefers Safari)
  • It does not implement authentication, logging or SSL encryption — it is meant to be used behind a nginx (or Apache) reverse proxy that will perform these functions.
  • It has no configuration file. There are just three variables to set at the top of the source file.
  • It does not implement the full server protocol, just the parts that are actually used by the extension today.
  • More controversially, it does not even implement persistence, keeping all data in RAM instead. Python running on Solaris is very reliable, and the expected uptime of the server is likely months on end. If the server fails, the Firefoxes will just have to perform a full sync and reconciliation. Fortunately, that has been much improved in Weave 0.6, so the cost is minimal. This could even be construed as a security feature, since there is no data on disk to be misplaced. It would take catastrophically losing all my browsers simultaneously to risk data loss. Short of California falling into the ocean, that’s not going to happen, and if it does, I probably have more pressing concerns…

The code could be extended fairly easily to lift these hypotheses, e.g. adding persistence or multiple user support using SQLite, PostgreSQL or MySQL.

Here is the server itself, weave_server.py:

#!/usr/local/bin/python
"""
  Based on tools/scripts/weave_server.py from
  http://hg.mozilla.org/labs/weave/

  do the Simplest Thing That Can Work: just enough to get by with Weave 0.6
  - SSL, authentication and loggin are done by nginx or other reverse proxy
  - no persistence, in case of process failure do a full resync
  - only one user. If you need more, create multiple instances on different
    ports and use rewrite rules to route traffic to the right one
"""

import sys, time, logging, socket, urlparse, httplib, pprint
try:
  import simplejson as json
except ImportError:
  import json
import wsgiref.simple_server

URL_BASE = 'https://your.server.name/'
#BIND_IP = ''
BIND_IP = '127.0.0.1'
DEFAULT_PORT = 8000

class HttpResponse:
  def __init__(self, code, content='', content_type='text/plain'):
    self.status = '%s %s' % (code, httplib.responses.get(code, ''))
    self.headers = [('Content-type', content_type),
                    ('X-Weave-Timestamp', str(timestamp()))]
    self.content = content or self.status

def JsonResponse(value):
  return HttpResponse(httplib.OK, value, content_type='application/json')

class HttpRequest:
  def __init__(self, environ):
    self.environ = environ
    content_length = environ.get('CONTENT_LENGTH')
    if content_length:
      stream = environ['wsgi.input']
      self.contents = stream.read(int(content_length))
    else:
      self.contents = ''

def timestamp():
  # Weave rounds to 2 digits and so must we, otherwise rounding errors will
  # influence the "newer" and "older" modifiers
  return round(time.time(), 2)

class WeaveApp():
  """WSGI app for the Weave server"""
  def __init__(self):
    self.collections = {}

  def url_base(self):
    """XXX should derive this automagically from self.request.environ"""
    return URL_BASE

  def ts_col(self, col):
    self.collections.setdefault('timestamps', {})[col] = str(timestamp())

  def parse_url(self, path):
    if not path.startswith('/0.5/') and not path.startswith('/1.0/'):
      return
    command, args = path.split('/', 4)[3:]
    return command, args

  def opts_test(self, opts):
    if 'older' in opts:
      return float(opts['older'][0]).__ge__
    elif 'newer' in opts:
      return float(opts['newer'][0]).__le__
    else:
      return lambda x: True

  # HTTP method handlers

  def _handle_PUT(self, path, environ):
    command, args = self.parse_url(path)
    col, key = args.split('/', 1)
    assert command == 'storage'
    val = self.request.contents
    if val[0] == '{':
      val = json.loads(val)
      val['modified'] = timestamp()
      val = json.dumps(val, sort_keys=True)
    self.collections.setdefault(col, {})[key] = val
    self.ts_col(col)
    return HttpResponse(httplib.OK)

  def _handle_POST(self, path, environ):
    try:
      status = httplib.NOT_FOUND
      if path.startswith('/0.5/') or path.startswith('/1.0/'):
        command, args = self.parse_url(path)
        col = args.split('/')[0]
        vals = json.loads(self.request.contents)
        for val in vals:
          val['modified'] = timestamp()
          self.collections.setdefault(col, {})[val['id']] = json.dumps(val)
        self.ts_col(col)
        status = httplib.OK
    finally:
      return HttpResponse(status)

  def _handle_DELETE(self, path, environ):
    assert path.startswith('/0.5/') or path.startswith('/1.0/')
    response = HttpResponse(httplib.OK)
    if path.endswith('/storage/0'):
      self.collections.clear()
    elif path.startswith('/0.5/') or path.startswith('/1.0/'):
      command, args = self.parse_url(path)
      col, key = args.split('/', 1)
      if not key:
        opts = urlparse.parse_qs(environ['QUERY_STRING'])
        test = self.opts_test(opts)
        col = self.collections.setdefault(col, {})
        for key in col.keys():
          if test(json.loads(col[key]).get('modified', 0)):
            logging.info('DELETE %s key %s' % (path, key))
            del col[key]
      else:
        try:
          del self.collections[col][key]
        except KeyError:
          return HttpResponse(httplib.NOT_FOUND)
    return response

  def _handle_GET(self, path, environ):
    if path.startswith('/0.5/') or path.startswith('/1.0/'):
      command, args = self.parse_url(path)
      return self.handle_storage(command, args, path, environ)
    elif path.startswith('/1/'):
      return HttpResponse(httplib.OK, self.url_base())
    elif path.startswith('/state'):
      return HttpResponse(httplib.OK, pprint.pformat(self.collections))
    else:
      return HttpResponse(httplib.NOT_FOUND)

  def handle_storage(self, command, args, path, environ):
    if command == 'info':
      if args == 'collections':
        return JsonResponse(json.dumps(self.collections.get('timestamps', {})))
    if command == 'storage':
      if '/' in args:
        col, key = args.split('/')
      else:
        col, key = args, None
      try:
        if not key: # list output requested
          opts = urlparse.parse_qs(environ['QUERY_STRING'])
          test = self.opts_test(opts)
          result = []
          for val in self.collections.setdefault(col, {}).itervalues():
            val = json.loads(val)
            if test(val.get('modified', 0)):
              result.append(val)
          result = sorted(result,
                          key=lambda val: (val.get('sortindex'),
                                           val.get('modified')),
                          reverse=True)
          if 'limit' in opts:
            result = result[:int(opts['limit'][0])]
          logging.info('result set len = %d' % len(result))
          if 'application/newlines' in environ.get('HTTP_ACCEPT', ''):
            value = '\n'.join(json.dumps(val) for val in result)
            return HttpResponse(httplib.OK, value,
                                content_type='application/text')
          else:
            return JsonResponse(json.dumps(result))
        else:
          return JsonResponse(self.collections.setdefault(col, {})[key])
      except KeyError:
        if not key: raise
        return HttpResponse(httplib.NOT_FOUND, '"record not found"',
                            content_type='application/json')

  def __process_handler(self, handler):
    path = self.request.environ['PATH_INFO']
    response = handler(path, self.request.environ)
    return response

  def __call__(self, environ, start_response):
    """Main WSGI application method"""

    self.request = HttpRequest(environ)
    method = '_handle_%s' % environ['REQUEST_METHOD']

    # See if we have a method called 'handle_METHOD', where
    # METHOD is the name of the HTTP method to call.  If we do,
    # then call it.
    if hasattr(self, method):
      handler = getattr(self, method)
      response = self.__process_handler(handler)
    else:
      response = HttpResponse(httplib.METHOD_NOT_ALLOWED,
                              'Method %s is not yet implemented.' % method)

    start_response(response.status, response.headers)
    return [response.content]

class NoLogging(wsgiref.simple_server.WSGIRequestHandler):
  def log_request(self, *args):
    pass

if __name__ == '__main__':
  socket.setdefaulttimeout(300)
  if '-v' in sys.argv:
    logging.basicConfig(level=logging.DEBUG)
    handler_class = wsgiref.simple_server.WSGIRequestHandler
  else:
    logging.basicConfig(level=logging.ERROR)
    handler_class = NoLogging
  logging.info('Serving on port %d.' % DEFAULT_PORT)
  app = WeaveApp()
  httpd = wsgiref.simple_server.make_server(BIND_IP, DEFAULT_PORT, app,
                                            handler_class=handler_class)
  httpd.serve_forever()

Here is the relevant fragment from my nginx configuration file:

# Mozilla Weave
location /0.5 {
  auth_basic            "Weave";
  auth_basic_user_file  /home/majid/web/conf/htpasswd.weave;
  proxy_pass            http://localhost:8000;
  proxy_set_header      Host $http_host;
}
location /1.0 {
  auth_basic            "Weave";
  auth_basic_user_file  /home/majid/web/conf/htpasswd.weave;
  proxy_pass            http://localhost:8000;
  proxy_set_header      Host $http_host;
}
location /1/ {
  auth_basic            "Weave";
  auth_basic_user_file  /home/majid/web/conf/htpasswd.weave;
  proxy_pass            http://localhost:8000;
  proxy_set_header      Host $http_host;
}

This code is hereby released into the public domain. You are welcome to use it as you wish. Just keep in mind that since it is reverse-engineered, it may well break with future releases of the Weave extension, or if Mozilla changes the server protocol.

Update (2009-10-03):

I implemented some minor changes for compatibility with Weave 0.7. The diff with the previous version is as follows:

--- weave_server.py~	Thu Sep  3 17:46:44 2009
+++ weave_server.py	Sat Oct  3 02:59:19 2009
@@ -65,8 +65,7 @@
     command, args = path.split('/', 4)[3:]
     return command, args

-  def opts_test(self, environ):
-    opts = urlparse.parse_qs(environ['QUERY_STRING'])
+  def opts_test(self, opts):
     if 'older' in opts:
       return float(opts['older'][0]).__ge__
     elif 'newer' in opts:
@@ -92,7 +91,7 @@
   def _handle_POST(self, path, environ):
     try:
       status = httplib.NOT_FOUND
-      if path.startswith('/0.5/') and path.endswith('/'):
+      if path.startswith('/0.5/'):
         command, args = self.parse_url(path)
         col = args.split('/')[0]
         vals = json.loads(self.request.contents)
@@ -113,7 +112,8 @@
       command, args = self.parse_url(path)
       col, key = args.split('/', 1)
       if not key:
-        test = self.opts_test(environ)
+        opts = urlparse.parse_qs(environ['QUERY_STRING'])
+        test = self.opts_test(opts)
         col = self.collections.setdefault(col, {})
         for key in col.keys():
           if test(json.loads(col[key]).get('modified', 0)):
@@ -142,10 +142,14 @@
       if args == 'collections':
         return JsonResponse(json.dumps(self.collections.get('timestamps', {})))
     if command == 'storage':
-      col, key = args.split('/')
+      if '/' in args:
+        col, key = args.split('/')
+      else:
+        col, key = args, None
       try:
         if not key: # list output requested
-          test = self.opts_test(environ)
+          opts = urlparse.parse_qs(environ['QUERY_STRING'])
+          test = self.opts_test(opts)
           result = []
           for val in self.collections.setdefault(col, {}).itervalues():
             val = json.loads(val)
@@ -155,6 +159,8 @@
                           key=lambda val: (val.get('sortindex'),
                                            val.get('modified')),
                           reverse=True)
+          if 'limit' in opts:
+            result = result[:int(opts['limit'][0])]
           logging.info('result set len = %d' % len(result))
           if 'application/newlines' in environ.get('HTTP_ACCEPT', ''):
             value = '\n'.join(json.dumps(val) for val in result)

Update (2009-11-17):

Weave 1.0b1 uses 1.0 as the protocol version string instead of 0.5 but is otherwise unchanged. I updated the script and nginx configuration accordingly.

Diminishing returns

I have an eight-core Nehalem Mac Pro. Most of these cores sit idle most of the time due to poorly written software that is not optimized for the post-Moore multicore world.

I am beginning to wonder if Intel’s transistor budget wouldn’t be better allocated to more SRAM cache instead of more cores. One SRAM bit uses up 4 transistors, the Xeon 5500 have 751 million transistors, of which 8Mx8x4 or 256 million are for the 8MB L3 cache. If the chip were brought down from quad-core to dual-core, that would allow doubling the cache. Many programs could run entirely from cache, including interpreters.

30 years after, the king of calculators rides again

In 1986, I purchased a Hewlett-Packard HP-15C scientific programmable calculator, for $120 or so. That was a lot of money back then, specially for a penniless high school student, but worth every penny. I lived in France at the time, and HP calculators cost roughly double the price there, so I waited for a vacation visit to my aunt in Los Angeles to get it. HP calculators are professional tools for engineers and you couldn’t find them at the local department store like TI trash, so I asked my aunt to mail order it for me prior to my visit. I still remember the excitement at finally getting it and putting it through its paces.

The HP-15C is long discontinued but I still keep mine as a prized heirloom, even though I have owned far more capable HPs over time (the HP-28C, HP-48SX, HP-200LX and more recently HP-35S and HP-33S) and given most of these away. The HP-15C’s financial cousin, the HP-12C is still in production today and has a tremendous cult following.

The reason for the HP Voyager series’ lasting power is many-fold:

  • Reverse Polish Notation (RPN), HP’s distinctive way of entering calculations. For instance, to calculate the area of a 2m radius circle, you would type 2 x2 π x instead of the more common algebraic (AOS) notation on TI or Casio calculators π x 2 x2 =. With practice, RPN is much more natural and efficient than algebraic notation. When I went from high school to college, the number of RPN users went from 2 (myself and a classmate who owned a HP-11C) to over 50%.
  • The ergonomics of the calculators are top-notch, from the landscape orientation to the inimitable HP keyboards with their firm and positive response.
  • They offered far superior functionality, like the HP-15C’s built-in integrator, equation solver, matrix and complex algebra, or the HP-12C’s financial equation solver.

The HP-15C offers the right balance of power and usability. The HP-48SX was far more powerful, but if you stopped using it for more than a couple months, you would completely forget how to use it. The more advanced functionality like symbolic integration is better performed on a Mac or PC using Mathematica or the like, in any case.

Unfortunately, Carly Fiorina gutted the HP calculator department in one of the more egregious of her blunders during her disastrous tenure as CEO of HP, outsourcing R&D and manufacturing from Corvallis, Oregon to China. HP has been trying to regain lost ground, but it is an uphill battle as TI has had ample time to entrench itself.

All this long exposition leads to the news HP has released an [In 1986, I purchased a Hewlett-Packard HP-15C scientific programmable calculator, for $120 or so. That was a lot of money back then, specially for a penniless high school student, but worth every penny. I lived in France at the time, and HP calculators cost roughly double the price there, so I waited for a vacation visit to my aunt in Los Angeles to get it. HP calculators are professional tools for engineers and you couldn’t find them at the local department store like TI trash, so I asked my aunt to mail order it for me prior to my visit. I still remember the excitement at finally getting it and putting it through its paces.

The HP-15C is long discontinued but I still keep mine as a prized heirloom, even though I have owned far more capable HPs over time (the HP-28C, HP-48SX, HP-200LX and more recently HP-35S and HP-33S) and given most of these away. The HP-15C’s financial cousin, the HP-12C is still in production today and has a tremendous cult following.

The reason for the HP Voyager series’ lasting power is many-fold:

  • Reverse Polish Notation (RPN), HP’s distinctive way of entering calculations. For instance, to calculate the area of a 2m radius circle, you would type 2 x2 π x instead of the more common algebraic (AOS) notation on TI or Casio calculators π x 2 x2 =. With practice, RPN is much more natural and efficient than algebraic notation. When I went from high school to college, the number of RPN users went from 2 (myself and a classmate who owned a HP-11C) to over 50%.
  • The ergonomics of the calculators are top-notch, from the landscape orientation to the inimitable HP keyboards with their firm and positive response.
  • They offered far superior functionality, like the HP-15C’s built-in integrator, equation solver, matrix and complex algebra, or the HP-12C’s financial equation solver.

The HP-15C offers the right balance of power and usability. The HP-48SX was far more powerful, but if you stopped using it for more than a couple months, you would completely forget how to use it. The more advanced functionality like symbolic integration is better performed on a Mac or PC using Mathematica or the like, in any case.

Unfortunately, Carly Fiorina gutted the HP calculator department in one of the more egregious of her blunders during her disastrous tenure as CEO of HP, outsourcing R&D and manufacturing from Corvallis, Oregon to China. HP has been trying to regain lost ground, but it is an uphill battle as TI has had ample time to entrench itself.

All this long exposition leads to the news HP has released an][1] (and others

hp15c

I benchmarked it by integrating the normal distribution (f LBL A x2 CHS ex 2 ENTER π x √x / RTN) between -3 and +3. On the original HP-15C, this takes about 34 seconds. On the iPhone emulator, it is near instantaneous. On the nonpareil emulator running on my octo-core Mac Pro, it’s more like a minute…