Scanner group test

TL:DR—avoid scanners with Contact Image Sensors if you care at all about color fidelity.

Vermeer it is not

After my abortive trial of the Colortrac SmartLF Scan, I did a comparative test of scanning one of my daughter’s A3-sized drawings on a number of scanners I had handy.

Scanner Sensor Scan
Colortrac SmartLF Scan CIS ScanLF.jpg
Epson Perfection Photo V500 Photo (manually stitched) CCD Epson_V500.jpeg
Epson Perfection V19 (manually stitched) CIS Epson_V19.jpg
Fujitsu ScanSnap S1500M (using a carrier sheet and the built-in stitching) CCD S1500M_carriersheet.jpg
Fujitsu ScanSnap SV600 CCD SV600.jpg
Fuji X-Pro2 with XF 35mm f/1.4 lens, mounted on a Kaiser RS2 XA copy stand with IKEA KVART 3-spot floor lamp (CCT 2800K, a mediocre 82 CRI as measured with my UPRtek CV600) CMOS X-Pro2.jpg

I was shocked by the wide variance in the results, as was my wife. This is most obvious in the orange flower on the right.


I scanned a swatch of the orange using a Nix Pro Color Sensor (it’s the orange square in the upper right corner of each scan in the comparison above). When viewed on my freshly calibrated NEC PA302W SpectraView II monitor, the Epson V500 scan is closest, followed by the ScanSnap SV600.

The two scanners using Contact Image Sensor (CIS) technology yielded dismal results. CIS are used in low-end scanners, and they have the benefit of low power usage, which is why the only USB bus-powered scanners available are all CIS models. CIS sensors begat the CMOS sensors used by the vast majority of digital cameras today, superseding CCDs in that application, I would not have expected such a gap in quality.

The digital camera scan was also quite disappointing. I blame the poor quality of the LEDs in the IKEA KVART three-headed lamp I used (pro tip: avoid IKEA LEDs like the plague, they are uniformly horrendous).

I was pleasantly surprised by the excellent performance of the S1500M document scanner. It is meant to be used for scanning sheaves of documents, not artwork, but Fujitsu did not skimp and used a CCD sensor element, and it shows.

Pro tip: a piece of anti-reflective Museum Glass or equivalent can help with curled originals on the ScanSnap SV600. I got mine from scraps at a framing shop. I can’t see a trace of reflections on the scan, unlike on the copy stand.

Update (2018-10-14):

Even more of a pro tip: a Japanese company named Bird Electron makes a series of accessories for the ScanSnap line, including a dust cover for the SV600 and the hilariously Engrish-named PZ-BP600 Book Repressor, essentially a sheet of 3mm anti-reflection coated acrylic with convenient carry handles. They are readily available on eBay from Japanese sellers.

Avery 22807 template for InDesign

The Avery 22807 2-inch circular stickers are a good alternative to Moo, PSPrint et al when you need a small quantity of stickers in a hurry. Unfortunately Avery has not seen it fit to provide usable InDesign templates as they do with some of their other sticker SKUs, only Microsoft Word, which is needless to say inadequate. A search for “Avery 22807 Indesign template” yielded some, but they have issues with missing linked PDF files.

I reverse-engineered the Microsoft template to build one of my own, with dimensions (including the tricky almost-but-not-quite square grid spaced at 5/8″ horizontally but 7/12″ vertically) to simplify “Step and Repeat…”.

I have only tested this with my InDesign CS6, not sure if it will work with older versions.

Avery 22807 2-inch circular labels.indt

On the bugginess of El Capitan

I never updated my home Mac Pro to El Capitan. To paraphrase Borges, each successive Apple OS release since Snow Leopard makes you long for the previous one. Unfortunately I have no choice but to run the latest OS X release on my work Macs as that is usually required to run the latest Xcode, itself required for the App Store.

I did not realize how bad El Capitan was until I upgraded my work iMac (27-inch 5K model) to Sierra last week. Previously, I would experience a mean time between crashes of around 3 days. I thought it was flaky hardware (the problems started from when I unboxed the computer), but couldn’t find the time to take it to the Genius Bar. I had also experienced the same problem with my old home 2009 Nehalem Mac Pro, which I had taken to the office, in fact that’s why I bought the iMac in the first place (and the first one I ordered had to go back because of defective pins in the RAM expansion slots). The Mac Pro had previously been rock-steady at home.

Since I upgraded to Sierra, I haven’t had a single crash. The only possible conclusion is that El Capitan bugs were to blame. The only thing unusual about this iMac is I upgraded the RAM from OWC, but the memory passes testing using Micromat’s TechTool.

I am not one to look at the Steve Jobs era with rosy-tinted glasses, OS X has never had the same level of stability as Solaris or even Linux, but Apple’s hardware and software quality has really gone to the dogs of late, something Lloyd Chambers dubs Apple Core Rot.

I am now starting to hedge my bets and am testing Ubuntu for my laptop computing needs, first by repurposing my 2008-vintage first-generation MacBook Air that is no longer supported by OS X anyway (works, but painfully slow) and soon with a shiny new HP Spectre on order.

Avoiding counterfeit goods on Amazon: mission impossible?

I mentioned previously that I seldom shop for electronics on any more, preferring B&H Photo whenever possible. I now have another reason: avoiding counterfeit goods.

My company boardroom is in an electromagnetic war zone—dozens of competing WiFi access points combined with electronic interference from the US-101 highway just outside make WiFi reception tenuous at best, and unusable more often than not. To work around this, we set up a wired Ethernet switch, and since most of our staff use MacBook Airs, Apple USB Ethernet adapters purchased from Amazon. When I side-graded from my 15″ Retina MacBook Pro to a much more portable 12″ Retina MacBook, I wasn’t able to connect using the dongle, and the name of the device was interspersed with Chinese characters. At first I thought it was an issue with my Satechi USB-C hub, but I experienced the same problems via a genuine Apple USB-C multiport adapter as well.

Eventually I figured out the Ethernet dongles were counterfeit. The packaging, while very similar to Apple’s, was just a tiny bit off, like amateurish margins between the Apple logo and the edges of the card. On the dongles themselves, the side regulatory disclosures sticker was inset, not flush with the body of the adapter.

Counterfeiting is a major problem. By some accounts, one third of all Sandisk memory cards worldwide are counterfeits. In some cases like chargers or batteries, your equipment could be at risk, or even your very life. The counterfeit adapters we purchased from Amazon did not come from Amazon themselves but from a third-party merchant participating in the Amazon marketplace. To Amazon’s credit, we returned them for a prompt, no questions asked refund even though we bought them over six months ago, but it is hard to believe Amazon is unaware of the problem rather than willfully turning a blind eye to it.

My first reaction was to tell our Office Manager to make sure to buy only from Amazon rather than third-party merchants (pro tip: including “amazon” in your Amazon search terms will do that in most cases). Unfortunately, that may not be enough. Amazon has a “fulfilled by Amazon” program for merchants where you ship your goods to them, and they handle warehousing and fulfillment. These “fulfilled by Amazon” items are also more attractive to Prime members. One option Amazon offers is Stickerless, commingled inventory where the items you send are put into a common bin. Amazon still has the ability to trace the provenance of the item through its inventory management, but for purposes of order fulfillment they will be handled just like Amazon’s own stock. Some categories like groceries and beauty products are excluded, but electronics are not.

The implications are huge: even if the vendor is Amazon itself, you cannot be sure that the item is not counterfeit. All the more reason to buy only from trustworthy, single-vendor sites like B&H, even if shipping is a bit slower.

Mac Pro first impressions

I received my late 2013 black cylinder Mac Pro last Monday. I ordered the 6-core model with D700 GPUs, since the higher-core models can’t Turbo boost to the full 3.9GHz the 4-core and 6-core ones can, and thus for most poorly-parallelized apps will underperform. I had to get a Promise Pegasus J4 with a pair of Samsung 840 Evo SSDs to hold my files, as I need nearly 2TB to do so and that is not available on the internal SSD, and it would be a shame to hobble this machine with spinning rust.

Some notes I have not seen in the many reviews sloshing around the Internet this far:

  • It is not actually black, rather a dark metallic gray, the color of hematite.
  • The TOSlink digital optical out is now capable of 192kHZ/24 bit audio, whereas the old Mac Pro was limited to 96 kHz. Unfortunately, it is very hard to find POF cables and DACs that can reliably sustain that data rate.
  • It is dead quiet compared to the old Mac Pro, and even my work iMac.
All in all, a remarkable engineering feat. A HP Z820 may have more memory capacity, expandability and total horsepower in its BMW-designed case, but Apple is the one pushing the envelope in terms of design.

Fixing Mac software update NSURLErrorDomain error -1012

Software Update for system components on my home Mac Pro has not worked in a while, and I have had to resort to manually downloading and applying updates. The updates just wouldn’t appear in the Mac App Store app where they normally should.

After upgrading to Mavericks, I finally figured out why. Instead of silently ignoring the updates, Mavericks displays a not-so-helpful error message “NSURLErrorDomain Error -1012”. On inspecting network traffic from the App Store app, I noticed it connects using TLS 1.2 to, then aborts. It then hit me – in 2011, after Comodo was hacked, apparently by elements affiliated with the Iranian government, I revoked the trust setting on their root certificates. The certificate for is signed by Comodo, and thus Software Update could no longer establish a secure connection to Apple and that’s why it was failing.

This is not the only time a Certificate Authority was hacked. Dutch CA Diginotar, which included the Dutch government among its clients, suffered a breach, apparently also involving Iran. Microsoft, Mozilla, Google and Apple promptly revoked Diginotar’s root CA certificates, which quickly led to the company going out of business. I guess Comodo is larger (the EFF calls them “too big to fail”) and better politically connected (it helps when you have people like Phillip Hallam-Baker on the payroll), and managed to elude the same punishment it richly deserved.

Apple should really step up its game and ditch a security provider which demonstrated incompetence at its alleged core competency, and I filed Radar bug report 15328323 to urge them to do so. In the meantime, the way to fix the error message is to temporarily reinstate trust in the Comodo root CA.

Update (2015-10-29)

At some point in the last 2 years they switched from Comodo to Symantex (probably 2014-04-13 when the current certificate was issued). Unfortunately, Symantec has its own problems.

Street sweeping reminders in iCal

Parking signSan Francisco sweeps streets twice a month in residential neighborhoods, and you will be fined if your car is parked on a street being swept. On my street, the schedule is the first and third Monday of each month, between 9am and 11am. I was trying to create reminders to myself in my calendar. Unfortunately, iCal does not have the ability to specify a recurring event with that definition.

No matter, Python to the rescue, the script below generates a year’s worth of reminders 12 hours before the event, in iCal vCalendar format. It does not correct for holidays, you will have to remove those yourself.

"""Idora street sweeping calendar - 1st and 3rd Mondays of the month 9am-11am"""
import datetime
Monday = 0
one_day = datetime.timedelta(1)
today =
year = today.year
month = today.month

def output(day):
  print """
SUMMARY:Idora street sweeping
""" % {
    'end': day.strftime('%Y%m%dT110000'),
    'start': day.strftime('%Y%m%dT090000')


for i in range(12):
  day =, month, 1)
  while day.weekday() != Monday:
    day += one_day
  output(day + 14 * one_day)
  month += 1
  if month > 12:
    month = 1
    year += 1



This is a popup the iOS6 App Store shows me when I try searching for a Google Maps app.


Scanning your iTunes library for DRM-infested books

Tor, the leading publisher for Science Fiction and Fantasy books, announced they would be doing away with DRM in their eBooks. The product pages for their books on iBooks now mention “At the publisher’s request, this title is being sold without Digital Rights Management software (DRM) applied”. I figured it would be a good idea to uncripple the many Tor eBooks I have in my collection.

I wrote a quick little Python script to scan my growing iBooks library for books that could be updated. The procedure is to delete the book from both iTunes and iPads, then download it anew (restarting iTunes is also needed after deleting). Apple keeps track of your purchases and will not charge you again.

#!/usr/bin/env python
import sys, os.path, glob, zipfile, platform, xml.etree.ElementTree

# publishers who have forsaken DRM
good = ['Tom Doherty']

if platform.mac_ver()[0] > '10.8':
  bookdir = os.path.expanduser('~/Library/Containers/')
  bookdir = os.path.expanduser('~/Music/iTunes/iTunes Music/Books')


ok =  '\033[1;32mDRM-free    \033[0m'
bad = '\033[1;31mDRM-infested\033[0m'

count = 0
salvageable = 0

def extract(meta):
  creator = ''
  status = ''
  pub = ''
  et = xml.etree.ElementTree.fromstring(meta)
    creator = et.findall('*{}creator')
    creator = creator[0].text
    title = et.findall('*{}title')
    title = title[0].text
    assert '!DOCTYPE plist' in meta
    next_tag = None
    for e in et[0].iter():
      if e.tag == 'key' and e.text in ('artistName', 'itemName'):
        next_tag = e.text
      if next_tag == 'artistName':
        creator = e.text
        next_tag = None
      elif next_tag == 'itemName':
        title = e.text
        next_tag = None
  pub = [x for x in good if x in meta]
  return creator, title, pub

def find_meta(file_list, opener):
  for m in file_list:
    if m.endswith('.opf') or m == 'iTunesMetadata.plist':
      meta = opener(m).read()
      return extract(meta)
for fn in glob.glob('*/*.epub'):
  status = ok
  suffix = ''
  if os.path.isdir(fn):
    suffix = '(directory)'
    if os.path.exists(fn + '/META-INF/encryption.xml'):
      status = bad
      count += 1
    meta = find_meta(os.listdir(fn), lambda x: open(fn + '/' + x))
    z = zipfile.ZipFile(fn)
      i = z.getinfo('META-INF/encryption.xml')
      status = bad
    except KeyError:
    meta = find_meta(z.namelist(),
  creator, title, pub = meta
  print status, fn, suffix
  print '\t', creator
  print '\t', title
  if status == bad and pub:
    print '\t\033[1;32mThis is published by', pub[0],
    print 'and could be re-downloaded DRM-free\033[0m'
    salvageable += 1

print count, 'books are DRM-infested'
print salvageable, 'could be cured'

Unfortunately, it seems like the DRM-stripping is still work in progress. Out of the Wheel of Time series, for instance, only the first one is now DRM-free on the iBooks store.

undr ~>
DRM-free     Books/0083D0AEC37E08453347DD12B1C6F980.epub
    Greg Bear
    Blood Music
DRM-free     Books/09178837756A4DFF8347EC377345A37B.epub
    Heinz Wittenbrink
    RSS and Atom
DRM-free     Books/0AD752E995042C7E12F11917AB58C6B8.epub
    Wes McKinney
    Python for Data Analysis
DRM-free     Books/14BDC66A99E878EC232FFAFA73B341EF.epub
    Fritz Leiber
    Swords and Deviltry-Fafhrd and the Gray Mouser-Book1
DRM-free     Books/15A1D7FE9B7D815C6FBE1A9A77D7143E.epub
    Glen Cook
    A Fortress in Shadow
DRM-free     Books/1793F9DE1319B96FDE7E36EB8A1BC961.epub
    Scalzi, John
    Old Man’s War
DRM-free     Books/1D08BE221E8BC8F2A371EFEDE55029AC.epub
    Ben Fry
    Visualizing Data
DRM-free     Books/24D6EC36CDEA0C1E8612CC61A89EA098.epub
    Node Cookbook
DRM-free     Books/29DA285F0051C431BD8BA3D1AEC5EAA6.epub
    Fritz Leiber
    The Swords of Lankhmar: Fafhrd and the Gray Mouser-Book 5
DRM-free     Books/2E88CD68DFD8408CD0E7C0ACB1E78714.epub
    Glen Cook
    A Cruel Wind: A Chronicle of the Dread Empire
DRM-free     Books/32996A9995040064818BAE4DFB66E92F.epub
    Kelly Link
    Magic for Beginners
DRM-free     Books/34D3CD13D47E5FEBC6DCF7EF011113BD.epub
    David Drake
    Lord of the Isles
DRM-infested Books/357298432.epub
    Iain M. Banks
    The Player of Games
DRM-infested Books/357311036.epub
    Iain M. Banks
    Use of Weapons
DRM-infested Books/357377857.epub
    Iain M. Banks
    Against a Dark Background
DRM-infested Books/357396585.epub
    Brent Weeks
    Night Angel: The Complete Trilogy
DRM-infested Books/357657026.epub
    Iain M. Banks
DRM-infested Books/357658374.epub
    Po Bronson
    NurtureShock: New Thinking About Children
DRM-infested Books/357662058.epub
    Iain M. Banks
    Consider Phlebas
DRM-infested Books/357669769.epub
    Iain M. Banks
DRM-infested Books/357914731.epub
    Herbert, Frank
    Dune Messiah
DRM-infested Books/357918110.epub
    Dalrymple, William
    City of Djinns
DRM-infested Books/357923567.epub
    Patrick Rothfuss
    The Name of the Wind
DRM-infested Books/357929995.epub
    Herbert, Frank
    Dune (40th Anniversary Edition)
DRM-infested Books/357969577.epub
    Herbert, Frank
    God Emperor of Dune
DRM-infested Books/357987322.epub
    Herbert, Frank
    Children of Dune
DRM-infested Books/357994537.epub
    Herbert, Frank
    Heretics of Dune
DRM-infested Books/357994652.epub
    William Dalrymple
    White Mughals: Love and Betrayal in Eighteenth-Century India
DRM-infested Books/357996119.epub
    Stross, Charles
DRM-infested Books/360601506.epub
    Ursula K. Le Guin
    The Dispossessed
DRM-infested Books/360609519.epub
    Greg Egan
    Schild’s Ladder
DRM-infested Books/360627712.epub
    Raymond E. Feist
    Rides a Dread Legion
DRM-infested Books/360627930.epub
    Neal Stephenson
DRM-infested Books/360628773.epub
    Raymond E. Feist
    At the Gates of Darkness
DRM-infested Books/360641088.epub
    Mihaly Csikszentmihalyi
DRM-free     Books/361491495.epub
    Basil Hall Chamberlain
    Aino Folk-Tales
DRM-free     Books/361494664.epub
    Poul William Anderson
    Industrial Revolution
DRM-free     Books/361523763.epub
    Lafcadio Hearn
    The Romance of the Milky Way / And Other Studies & Stories
DRM-free     Books/361527545.epub
    When William Came
DRM-free     Books/361539032.epub
    The Chronicles of Clovis
DRM-free     Books/361557387.epub
    Reginald in Russia and other sketches
DRM-free     Books/361557834.epub
    Sir Arthur Conan Doyle
    The Adventure of the Dying Detective
DRM-free     Books/361559391.epub
    Sir Arthur Conan Doyle
    The Valley of Fear
DRM-free     Books/361560694.epub
    Lafcadio Hearn
    Chita: a Memory of Last Island
DRM-free     Books/361561399.epub
    The Analects of Confucius (from the Chinese Classics)
DRM-free     Books/361562678.epub
    The Toys of Peace, and other papers
DRM-free     Books/361562764.epub
    Sir Arthur Conan Doyle
    The Memoirs of Sherlock Holmes
DRM-free     Books/361564075.epub
    Poul William Anderson
    The Burning Bridge
DRM-free     Books/361564898.epub
    Henry David Thoreau
DRM-free     Books/361565201.epub
    Beasts and Super-Beasts
DRM-free     Books/361565806.epub
    Isaac Asimov
DRM-free     Books/361572327.epub
    Lafcadio Hearn
    Kokoro / Japanese Inner Life Hints
DRM-free     Books/361573126.epub
    Sir Arthur Conan Doyle
    Through the Magic Door
DRM-free     Books/361575882.epub
    Sir Arthur Conan Doyle
    Tales of Terror and Mystery
DRM-free     Books/361578744.epub
    Lafcadio Hearn
    In Ghostly Japan
DRM-free     Books/361588265.epub
    Lafcadio Hearn
    Books and Habits from the Lectures of Lafcadio Hearn
DRM-free     Books/361673695.epub
    E. C. Babbitt
    More Jataka Tales
DRM-free     Books/361686559.epub
    Poul William Anderson
DRM-free     Books/361713863.epub
    Sir Arthur Conan Doyle
    The Return of Sherlock Holmes
DRM-free     Books/361721797.epub
    Sir Arthur Conan Doyle
    The Adventure of the Cardboard Box
DRM-free     Books/361725352.epub
    The Unbearable Bassington
DRM-free     Books/361725959.epub
    Sir Arthur Conan Doyle
    The Adventure of Wisteria Lodge
DRM-free     Books/361726975.epub
DRM-free     Books/361727237.epub
    Sir Arthur Conan Doyle
    The Adventure of the Red Circle
DRM-free     Books/361732286.epub
    Poul William Anderson
    The Valor of Cappen Varra
DRM-free     Books/361736043.epub
    Lafcadio Hearn
    Kwaidan: Stories and Studies of Strange Things
DRM-free     Books/361741563.epub
    Lafcadio Hearn
    Japan: an Attempt at Interpretation
DRM-free     Books/361743007.epub
    Ambrose Bierce
    The Devil’s Dictionary
DRM-free     Books/361743953.epub
    Sir Arthur Conan Doyle
    The Adventure of the Devil’s Foot
DRM-free     Books/361744178.epub
    Sir Arthur Conan Doyle
    His Last Bow
DRM-free     Books/361745602.epub
    Poul William Anderson
    The Sensitive Man
DRM-infested Books/362435686.epub
    Ansary, Tamim
    Destiny Disrupted
DRM-infested Books/366773380.epub
    Esslemont, Ian C. C.
    Return of the Crimson Guard
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/373338999.epub
    Jordan, Robert
    The Path of Daggers
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-free     Books/375554215.epub
    Orson Scott Card
    The Lost Gate
DRM-infested Books/376217648.epub
    Steven Erikson
    Reaper’s Gale
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/376227359.epub
    Jordan, Robert
    Winter’s Heart
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/376227401.epub
    Jordan, Robert
    Crossroads of Twilight
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/376227406.epub
    Jordan, Robert
    Knife of Dreams
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/376227409.epub
    Sanderson, Brandon
    The Gathering Storm
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/376227423.epub
    Jordan, Robert
    New Spring
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-free     Books/376231110.epub
    Cook, Glen
    Surrender to the Will of the Night
DRM-infested Books/376231528.epub
    Robert Jordan and Brandon Sanderson
    Towers of Midnight
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/378317076.epub
    Jordan, Robert
    A Crown of Swords
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/378317808.epub
    Robert Jordan
    Lord of Chaos
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-free     Books/379451459.epub
    Le Guin, Ursula K.
    Word for World is Forest, The
DRM-free     Books/37EC6E895E6BD70BEB48D9F1553D608E.epub
    Eben Hewitt
    Cassandra: The Definitive Guide
DRM-free     Books/380490608.epub
    Jordan, Robert
    The Eye of the World
DRM-free     Books/380494444.epub
    Asimov, Isaac
    The End of Eternity
DRM-infested Books/381497257.epub
    Harold McGee
    On Food and Cooking, The Science and Lore of the Kitchen
DRM-infested Books/381622032.epub
    Look to Windward
DRM-infested Books/381683084.epub
    Ursula K. Le Guin
DRM-infested Books/381935940.epub
    Richard Adams
    Watership Down
DRM-infested Books/382674388.epub
    Steven Erikson
    Dust of Dreams
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/383912791.epub
    Steven Erikson
    Bauchelain and Korbal Broach
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/385975662.epub
    Jordan, Robert
    The Dragon Reborn
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/385981104.epub
    Steven Erikson
    The Bonehunters
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/385981116.epub
    Steven Erikson
    Midnight Tides
DRM-free     Books/385982966.epub
    Brust, Steven
    To Reign in Hell
DRM-infested Books/385987858.epub
    Steven Erikson
    Gardens of the Moon
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/385989170.epub
    Steven Erikson
    House of Chains
DRM-infested Books/385992628.epub
    Jordan, Robert
    The Fires of Heaven
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/385992927.epub
    Steven Erikson
    Toll the Hounds
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/385992930.epub
    Jordan, Robert
    The Great Hunt
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/385998417.epub
    Jordan, Robert
    The Shadow Rising
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/386016540.epub
    Esslemont, Ian C. C.
    Night of Knives
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/388403394.epub
    Steven Erikson
    The Crippled God
DRM-infested Books/389191300.epub
    Iain M. Banks
    Surface Detail
DRM-infested Books/390877859.epub
    Loewen, James W.
    Lies My Teacher Told Me
DRM-infested Books/393310992.epub
    Erikson, Steven
    Memories of Ice
DRM-infested Books/394745271.epub
    Steven Erikson
    Deadhouse Gates
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-free     Books/394745833.epub
    Walton, Jo
    Among Others
DRM-free     Books/395536306.epub
    Sir Arthur Conan Doyle
    The Adventures of Sherlock Holmes
DRM-free     Books/395537209.epub
    Sir Arthur Conan Doyle
    The Sign of the Four
DRM-free     Books/395539542.epub
    Sir Arthur Conan Doyle
    A Study in Scarlet
DRM-free     Books/395540660.epub
    Sir Arthur Conan Doyle
    The Hound of the Baskervilles
DRM-free     Books/395686685.epub
    Dante Alighieri
    Divine Comedy, Longfellow’s Translation, Complete
DRM-free     Books/395688318.epub
    Edgar Rice Burroughs
    A Princess of Mars
DRM-free     Books/395688375.epub
    Lafcadio Hearn
    Glimpses of an Unfamiliar Japan / First Series
DRM-infested Books/395926792.epub
    Fukuyama, Francis
    Origins of Political Order
DRM-infested Books/396269736.epub
    Herbert, Frank
    Chapterhouse: Dune
DRM-infested Books/398283114.epub
    Rothfuss, Patrick
    The Wise Man’s Fear
DRM-free     Books/3A5FBC58E821CFDF15C8C4E85657481E.epub
    Jon Hicks
    The Icon Handbook
DRM-free     Books/410943153.epub
    Edwin A. Abbott (A Square)
    Flatland: A Romance of Many Dimensions
DRM-free     Books/413463878.epub
    Brust, Steven
DRM-free     Books/418293515.epub
    Heinlein, Robert A.
    Glory Road
DRM-infested Books/419950945.epub
    Isaac Asimov
DRM-infested Books/419950970.epub
    Isaac Asimov
    Foundation and Empire
DRM-infested Books/419950976.epub
    Isaac Asimov
    Second Foundation
DRM-infested Books/419968238.epub
    Scott Lynch
    The lies of Locke Lamora
DRM-infested Books/419968784.epub
    Scott Lynch
    Red Seas Under Red Skies
DRM-infested Books/420037362.epub
    Kim Stanley Robinson
    The Years of Rice and Salt
DRM-infested Books/420281728.epub
    Richard Wiseman
    59 Seconds: Think a Little, Change a Lot
DRM-infested Books/420445771.epub
    Isaac Asimov
    Foundation’s Edge
DRM-infested Books/420446058.epub
    Isaac Asimov
    Foundation and Earth
DRM-infested Books/420725428.epub
    Mike Resnick
    Kirinyaga: A Fable of Utopia
DRM-infested Books/421025353.epub
    William Dalrymple
    The Last Mughal
DRM-free     Books/421124117.epub
    Brust, Steven
    The Desecrator
DRM-infested Books/422530144.epub
    Max Barry
    Machine Man
DRM-free     Books/422718511.epub
    Apple Inc.
    Mac Integration Basics
DRM-free     Books/426914658.epub
    Brust, Steven
    Five Hundred Years After
DRM-free     Books/428235697.epub
    Vinge, Vernor
    A Fire Upon The Deep
DRM-infested Books/429173089.epub
    Ursula K. Le Guin
    The Other Wind
DRM-infested Books/429173713.epub
    Ursula K. Le Guin
    Tales from Earthsea
DRM-infested Books/429699133.epub
    Rajaniemi, Hannu
    The Quantum Thief
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/431617578.epub
    Walter Isaacson
    Steve Jobs
DRM-infested Books/432519291.epub
    Susan Weinschenk
    100 Things: Every Designer Needs to Know About People
DRM-infested Books/434509014.epub
    Stross, Charles
    Rule 34
DRM-free     Books/434522188.epub
    Larry Niven, Jerry Pournelle
    The Mote In God’s Eye
DRM-free     Books/434811509.epub
    Asher, Neal
DRM-infested Books/436646026.epub
    Neal Stephenson
DRM-infested Books/436691174.epub
    Julia Child
    Mastering the Art of French Cooking
DRM-infested Books/443149884.epub
    Daniel Kahneman
    Thinking, Fast and Slow
DRM-infested Books/446155927.epub
    Esslemont, Ian C. C.
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-free     Books/447591195.epub
    Asher, Neal
    The Skinner
DRM-infested Books/454252718.epub
    William B. Norton
    The Internet Peering Playbook: Connecting to the Core of the Internet
DRM-infested Books/455525627.epub
    Amar Chitra Katha
    Birbal The Genius
DRM-infested Books/458461362.epub
    Pamela Druckerman
    Bringing Up Bebe
DRM-free     Books/45B90418E467D479DCDDF23B932C648C.epub
    Douglas Crockford
    JavaScript: The Good Parts
DRM-infested Books/460822066.epub (directory)
    Scott Lynch
    The Republic of Thieves
DRM-free     Books/465679A557523FDB836005CF4BB9380E.epub
    Scott Berkun
DRM-infested Books/479594044.epub
    Saladin Ahmed
    Throne of the Crescent Moon
DRM-infested Books/479717436.epub
    David Crist
    The Twilight War: The Secret History of America’s Thirty-Year Conflict with Iran
DRM-infested Books/479771801.epub
    William Dalrymple
    In Xanadu
DRM-infested Books/489957500.epub
    Bruce Schneier
    Liars and Outliers
DRM-infested Books/491186678.epub
    James Blish
    Cities in Flight
DRM-infested Books/491668459.epub
    Neal Asher
    Shadow of the Scorpion
DRM-infested Books/491669284.epub
    Glen Cook
    A Matter of Time
DRM-infested Books/491669288.epub
    Glen Cook
DRM-free     Books/492199230.epub
    Frederik Pohl
    The Tunnel Under the World
DRM-free     Books/492199569.epub
    Frederik Pohl
    The Knights of Arthur
DRM-infested Books/494939678.epub
    Esslemont, Ian C. C.
    Orb Sceptre Throne
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/498634992.epub
    Charles Stross
    The Apocalypse Codex
DRM-infested Books/499392787.epub
    Daniel Suarez
    Kill Decision
DRM-free     Books/4AE7DDA54BEEFCD65157927546B18063.epub
    Roberto Ierusalimschy
    Programming in Lua 2ed
DRM-free     Books/4DCFF682728B765A1CE221F3D7C21536.epub
    Glen Cook
    Starfishers Volume 3: Stars’ End
DRM-free     Books/501278407.epub
DRM-free     Books/501758197.epub
    David Brin
DRM-infested Books/501758516.epub
    Scalzi, John
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/503019669.epub
    J.R.R. Tolkien
    The Lord of the Rings
DRM-infested Books/503153300.epub
    J.R.R. Tolkien
    Tales from the Perilous Realm
DRM-infested Books/503154129.epub
    J. R. R. Tolkien and Christopher Tolkien
    The Book of Lost Tales, Part One
DRM-infested Books/503154991.epub
    J. R. R. Tolkien and Christopher Tolkien
    The Book of Lost Tales, Part Two
DRM-infested Books/503155327.epub
    J.R.R. Tolkien
    The Children of Húrin
DRM-infested Books/503163148.epub
    J.R.R. Tolkien
    The Hobbit Deluxe
DRM-infested Books/503164678.epub
    J.R.R. Tolkien
    The Silmarillion
DRM-infested Books/503167303.epub
    J.R.R. Tolkien
    Unfinished Tales of Númenor and Middle-earth
DRM-infested Books/504209078.epub
    Isaac Asimov
    Prelude to Foundation
DRM-infested Books/504371982.epub
    Iain M. Banks
    The Hydrogen Sonata
DRM-free     Books/511060740.epub
    Frederik Pohl
    The Hated
DRM-free     Books/511143617.epub
    Frederik Pohl
    The Day of the Boomer Dukes
DRM-free     Books/511252896.epub
    Frederik Pohl
DRM-free     Books/513357605.epub
    Hannu Rajaniemi
    The Fractal Prince
DRM-free     Books/513868CF0AA46D293EE27F74BC399760.epub
    Jerry Pournelle
    West of Honor
DRM-infested Books/520233773.epub
    Nate Silver
    The Signal and the Noise
DRM-free     Books/520897403.epub
    Hugh Howey
    Wool Omnibus
DRM-free     Books/525170910.epub
    Cory Doctorow and Charles Stross
    The Rapture of the Nerds
DRM-infested Books/526136048.epub (directory)
    Hetty van de Rijt & Frans Plooij
    The Wonder Weeks
DRM-infested Books/529020127.epub
    Neal Asher
    The Departure
DRM-infested Books/529424632.epub
    Guy Gavriel Kay
    The Lions of Al-Rassan
DRM-infested Books/536312376.epub
    Glen Cook
    Garrett for Hire
DRM-free     Books/537023027.epub
    Erikson, Steven
    Forge of Darkness
DRM-infested Books/541673159.epub
    Ursula K. Le Guin
    The Tombs of Atuan
DRM-infested Books/541673162.epub
    Ursula K. Le Guin
    The Farthest Shore
DRM-infested Books/546126326.epub
    Jan Morris
DRM-infested Books/551241606.epub
    Ursula K. Le Guin
    A Wizard of Earthsea
DRM-infested Books/551567785.epub
    Murray R. Spiegel, PhD
    Schaum’s Outline Mathematical Handbook of Formulas and Tables, Fourth Edition
DRM-infested Books/551747038.epub
    Chris Hedges and Joe Sacco
    Days of Destruction Days of Revolt v2b
DRM-infested Books/552144691.epub
    Tamim Ansary
    Games without Rules
DRM-free     Books/553878102.epub
    Charles Stross
    A Tall Tail
DRM-infested Books/563408849.epub
    Iain Banks
DRM-infested Books/568731449.epub
    William Dalrymple
    Return of a King: The Battle for Afghanistan, 1839-42
DRM-infested Books/569232538.epub
    Neil Gaiman
    The Ocean at the End of the Lane
DRM-free     Books/571678152.epub
    Glen Cook
    The Return of the Black Company
DRM-free     Books/571678945.epub
    Glen Cook
    The Many Deaths of the Black Company
DRM-free     Books/573656304.epub
    Glen Cook
    Chronicles of the Black Company
DRM-free     Books/573656441.epub
    Glen Cook
    The Books of the South
DRM-free     Books/576233114.epub
    Jack Vance
    Demon Princes
DRM-infested Books/578851675.epub
    Zilpha Keatley Snyder
    Below The Root
DRM-infested Books/580642602.epub
    Max Barry
DRM-infested Books/588794444.epub
    Charles Stross
    Neptune’s Brood
DRM-free     Books/5CCB889F91585202637A3C5FBFD8409F.epub
    Glen Cook
    Starfishers-The Starfishers Trilogy Volume II
DRM-infested Books/600938002.epub
    Gardner Dozois
    The Year’s Best Science Fiction: Thirtieth Annual Collection
DRM-free     Books/606232503.epub
    Ian C. Esslemont
    Blood and Bone
DRM-free     Books/610920977.epub
    Robert Jordan and Brandon Sanderson
    A Memory of Light
DRM-free     Books/6122FF30560866BD75257E6CCC264371.epub
    Fritz Leiber
    Swords and Ice Magic-Fafhrd and the Gray Mouser-Book 6
DRM-infested Books/619483561.epub
    Raymond E. Feist
    Magician’s End
DRM-free     Books/622837311.epub
    Le Comte De  Lautréamont
    Les chants de Maldoror
DRM-free     Books/62497334A4189B1D00E9BDFD95724E2E.epub
    Fritz Leiber
    The Knight and Knave of Swords-Fafhrd and the Gray Mouser-Book 7
DRM-infested Books/645571245.epub
    Iain Banks
    The Quarry
DRM-free     Books/647688922.epub
    Steven Brust and Skyler White
    The Incrementalists
DRM-infested Books/651331715.epub
    Susan Crawford
    Captive Audience
DRM-infested Books/654456347.epub
    Iain Banks
    The Wasp Factory
DRM-infested Books/662310218.epub (directory)
    Various Authors
    Star Wars: Empire Volume 3 – The Imperial Perspective
DRM-infested Books/662310219.epub
    Paul Gulacy
    Star Wars: Crimson Empire
DRM-infested Books/664297397.epub (directory)
    Various Authors
    Star Wars: Empire, Vol. 4: The Heart of the Rebellion
DRM-infested Books/664343525.epub (directory)
    Paul Chadwick, Doug Wheatley & Tomás Giorello
    Star Wars: Empire, Vol. 2: Darklighter
DRM-infested Books/664894567.epub
    John Ostrander
    Star Wars: Dawn of the Jedi Volume 1—Force Storm
DRM-infested Books/664910993.epub (directory)
    Scott Allie, Ryan Benjamin & Brian Horton
    Star Wars: Empire Vol. 1
DRM-free     Books/666772E4C21E2017E71C10F1990840BC.epub
    Pieter Hintjens
    ZeroMQ - Connecting your Code
DRM-infested Books/674224604.epub (directory)
    Various Authors
    Star Wars: Empire, Vol. 5: Allies and Adversaries
DRM-infested Books/674226603.epub (directory)
    Thomas Andrews, Scott Allie & Various Authors
    Star Wars: Empire, Vol. 6: In the Shadows of Their Fathers
DRM-free     Books/697938901.epub
    Charles Stross
    Equoid: A Laundry Novella
DRM-free     Books/6BBD61A64A46FFFF4AE7C5FCEB9CFCEE.epub
    David Drake
DRM-free     Books/71C79A253586282ABE73C2975237EB08.epub
    Glen Cook
    Sung in Blood
DRM-free     Books/73BC5DD0E51611BDC359CBAB48CC203F.epub (directory)
    Crane, Stephen
    The Red Badge of Courage
DRM-free     Books/7A0D1AEE343638A8A3769DABB90CD4CB.epub
    Clay A. Johnson
    The Information Diet
DRM-free     Books/7B46914B55481714DED3D711288978FA.epub
    Glen Cook
    Shadowline-The Starfishers Trilogy I
DRM-free     Books/809DCC75FE60E749458CB27636EE6777.epub
    Mercedes Lackey
    The Secret World Chronicle
DRM-free     Books/8350F5DFECCD6AA88714400FDF4F6831.epub
    François de La Rochefoucauld
    Réflexions ou Sentences et Maximes Morales
DRM-free     Books/853BD1C65277D276BA09E04EBFEB73EF.epub
    PostgreSQL 9 Administration Cookbook
DRM-free     Books/85907063B2351E91C6E7F5052C090BFD.epub
    PostgreSQL 9.0 High Performance
DRM-free     Books/8663E7BF735C06318FF450532A67F1C2.epub
    Glen Cook
    Passage at Arms
DRM-free     Books/8690D69995483C5D2DF6AD38BE53C1D7.epub
    Paolo Bacigalupi
    The Windup Girl - Second Electronic Edition
DRM-free     Books/8993920EF1C0E8FE8CB47A20BD955F53.epub
    Fritz Leiber
    Swords Against Death-Fafhrd and Gray Mouser-Book 2
DRM-free     Books/8DC6ED0A99161EBB516E370A60ED1121.epub
    Jonathan Zdziarski
    Hacking and Securing iOS Applications
DRM-free     Books/8DD0D22CD05B0B8D52DF9DB93FC8616B.epub
    Fritz Leiber
    Swords in the Mist-Fafhrd And the Gray Mouser-Book 3
DRM-free     Books/912CB8B110736684549EAC4FC36665AB.epub
    Neil Gaiman and Dave McKean
    Signal to Noise
DRM-free     Books/93061FDFD6EEC01FD8CE4295A049C97C.epub
    Cory Doctorow
DRM-free     Books/9592D5001632B94D1FEFC98B3A40E049.epub
    Kelly Link
    Stranger Things Happen
DRM-free     Books/990CA5799084407151488A7C563DF269.epub
    Tom Hughes-Croucher
    Node: Up and Running
DRM-free     Books/9CCCA9E217602948B84BE9A7A21C2753.epub
    Mike Resnick
    Birthright: The Book of Man
DRM-free     Books/9DA2C8D7E941C37FA83192E5849AA850.epub
    Q. Ethan McCallum
    Parallel R
DRM-free     Books/A7DB557515983DB67F74E2BE351FC319.epub
    Glen Cook
    Reap the East Wind
DRM-free     Books/B18AB85267B815E540C7E370F8D97726.epub
    Lars George
    HBase: The Definitive Guide
DRM-free     Books/B1A586D24AE5F2450010F8664F8E059D.epub
    Cory Doctorow
    Pirate Cinema
DRM-free     Books/B562B5D74F8677C946B0A6F81EB34F1B.epub
    Gotthold Ephraim Lessing
    Nathan the Wise; a dramatic poem in five acts
DRM-free     Books/BE42D426242836AA171539B7415732E6.epub
    Glen Cook
    The Swordbearer
DRM-free     Books/BE80AD93781746E45CF95607A7BEE687.epub
    Charlie Stross
    Bit Rot
DRM-free     Books/C39FBC59673713A57D404D62BE85C4DC.epub
    Lauren Beukes
    Zoo City
DRM-free     Books/C81CBBD470EFDBC62359BDAD12FDF551.epub
    Thomas Hobbes
DRM-free     Books/D17DC1A83BD90AFB24055A01831BFAB4.epub
    Glen Cook
    The Dragon Never Sleeps
DRM-free     Books/DA8FBAB386EC503A0EF21E481D214521.epub
    David Flanagan
    JavaScript: The Definitive Guide
DRM-free     Books/DBAD0DE7A4305B90F6AC33C673FE68E8.epub
    Glen Cook
    A Path to Coldness of Heart
DRM-free     Books/DBCCB54B140D1158250B24B7E3E81B63.epub
    Scott Chacon
    Pro Git
DRM-free     Books/DF6C56F9A8719294B13BA91BED5E1667.epub
    Mike Resnick
DRM-free     Books/E8D76FDD351E68D92AE4A3F3AEA7EC6A.epub
    Paolo Bacigalupi
    Pump Six and Other Stories
DRM-free     Books/EEE65A68378C2510E18DF24CD767AC9A.epub
    Fritz Leiber
    Swords Against Wizardry-Fafhrd and the Gray Mouser-Book 4
DRM-free     Books/F3BC14466A16C96CCD8FFE00DCCF8147.epub
    Glen Cook
    An Empire Unacquainted With Defeat
DRM-free     Books/F534835595041374B814D151A633E69D.epub
    Peter Watts
DRM-free     Books/F8BF760284ADC800B290DCA6D8EA7EF2.epub
    Ben Klemens
    21st Century C
DRM-free     Books/FE8C57863E40B98CB732FEE4BFDB60BB.epub
    Glen Cook
    An Ill Fate Marshalling
8 books are DRM-infested
26 could be cured

Update (2013-11-06):

OS X 10.9 Mavericks and the new iBooks app changed the location of the iBooks directory, I changed my script accordingly (and made it adjust depending on which OS version you have). Also, the file names have changed and no longer embed author and title, so I am extracting them from the XML metadata files.

Fixing Microsoft Office Mac 2011 font woes

This post may be of use to people experiencing the same problem I had. When I opened PowerPoint slides sent by a coworker, I was getting blank slides. Attempting to switch to outline view would crash PowerPoint. A search for possible solutions yielded inconclusive posts or dead-ends.

After much twiddling, it turns out the problem was with a misconfigured Arial font. Microsoft in its great wisdom installs a copy of Arial that conflicts with the system-supplied one, as explained in Kurt Lang’s must-read article on OS X font management, but that was not the root cause of the problem.

Arial was not appearing at all in the PowerPoint font menu, and opening the slides in Keynote yielded a font warning saying non-existent Arial was replaced by Arial Narrow. Clearly PowerPoint is much less resilient than Keynote to missing fonts (crashing in outline view mode is inexcusable, but that’s Microsoft coding sloppiness for you). If Arial is so vital, it should have been included as a resource in the application bundle itself, but I digress.

Font Book did show Arial as installed, with only a single copy after I purged the conflicting fonts as per Kurt Lang’s recommendations. Validating Arial in Font Book yielded a warning about duplicate fonts, but with no indication of where the duplicate may be, since Font Book itself did not know of any other instance.

It turns out I had an old pre-OpenType copy of Arial in my ~/Library/Fonts folder that was causing a conflict. It was functional enough to be picked up as conflicting and disabling Arial in PowerPoint and Keynote, but not functional enough to be listed by Font Book. I am not sure how it got there, possibly from an older version of Office and transferred over many machine upgrades. Removing the file fixed the problem. of blank slides and the crash in switching to outline view.

Of course, if you care at all about typography, you should use Helvetica rather than an inferior ersatz like Arial, but Arial is so prevalent in the taste-impaired Windows world that one cannot escape its gaucheness entirely.


Solving Mac freezes due to ocspd crashes

A public service announcement for anyone experiencing the same problem and who may google for a solution.

A couple of days ago my work iMac started experiencing intermittent freezing, and very slow searches in Rebuilding my Spotlight index overnight using sudo mdutil -Eav did not help. I started suspecting problems with recent versions of Chrome, or iTunes 10.5 Beta 7, but quitting those apps did not help either.

After running the, I noticed the following messages in my logs:

2011-09-14 17:39:20[1]    ([3197]) Job appears to have crashed: Bus error
2011-09-14 17:39:21[3201]    2011-09-14 17:39:21.325 ReportCrash[3201:2903] Saved crash report for ocspd[3197] version ??? (???) to /Library/Logs/DiagnosticReports/ocspd_2011-09-14-173921_localhost.crash
2011-09-14 17:50:29[1]    ([3269]) Job appears to have crashed: Bus error
2011-09-14 17:50:29[3270]    2011-09-14 17:50:29.964 ReportCrash[3270:2903] Saved crash report for ocspd[3269] version ??? (???) to /Library/Logs/DiagnosticReports/ocspd_2011-09-14-175029_localhost.crash
2011-09-14 17:50:45[1]    ([3271]) Job appears to have crashed: Bus error
2011-09-14 17:50:45[3270]    2011-09-14 17:50:45.117 ReportCrash[3270:2807] Saved crash report for ocspd[3271] version ??? (???) to /Library/Logs/DiagnosticReports/ocspd_2011-09-14-175045_localhost.crash

Looking at those crash dumps did not yield very useful information, just some blather like this:

Process:         ocspd [3197]
Path:            /usr/sbin/ocspd
Identifier:      ocspd
Version:         ??? (???)
Code Type:       X86-64 (Native)
Parent Process:  launchd [1]

Date/Time:       2011-09-14 17:39:19.339 -0700
OS Version:      Mac OS X 10.6.8 (10K549)
Report Version:  6

Exception Type:  EXC_BAD_ACCESS (SIGBUS)
Exception Codes: 0x000000000000000a, 0x000000010009b210
Crashed Thread:  0  Dispatch queue:

Thread 0 Crashed:  Dispatch queue:
0              0x00007fff87b59d0b Security::ReadSection::at(unsigned int) const + 25
1              0x00007fff87b59172 Security::DbVersion::open() + 62
2              0x00007fff87b58cc1 Security::DbVersion::DbVersion(Security::AppleDatabase const&, Security::RefPointer const&) + 179
3              0x00007fff87b587ce Security::DbModifier::getDbVersion(bool) + 330
4              0x00007fff87b58675 Security::DbModifier::openDatabase() + 33
5              0x00007fff87b582b9 Security::Database::_dbOpen(Security::DatabaseSession&, unsigned int, Security::AccessCredentials const*, void const*) + 221
6              0x00007fff87b576c1 Security::DatabaseManager::dbOpen(Security::DatabaseSession&, Security::DbName const&, unsigned int, Security::AccessCredentials const*, void const*) + 77
7              0x00007fff87b575a3 Security::DatabaseSession::DbOpen(char const*, cssm_net_address const*, unsigned int, Security::AccessCredentials const*, void const*, long&) + 285
8              0x00007fff87b6b294 cssm_DbOpen(long, char const*, cssm_net_address const*, unsigned int, cssm_access_credentials const*, void const*, long*) + 108
9              0x00007fff87b6ae3a CSSM_DL_DbOpen + 106
10  ocspd                           0x0000000100006ad9 0x100000000 + 27353
11  ocspd                           0x0000000100006cab 0x100000000 + 27819
12  ocspd                           0x0000000100001f68 0x100000000 + 8040
13  ocspd                           0x00000001000176ed 0x100000000 + 95981
14  ocspd                           0x000000010001787b 0x100000000 + 96379
15  ocspd                           0x0000000100017e4f 0x100000000 + 97871
16  ocspd                           0x0000000100004613 0x100000000 + 17939
17  ocspd                           0x0000000100001d48 0x100000000 + 7496

Thread 0 crashed with X86 Thread State (64-bit):
  rax: 0x000000010008e000  rbx: 0x000000010008e000  rcx: 0x00007fff88f0d47a  rdx: 0x000000000000d210
  rdi: 0x0000000100115478  rsi: 0x000000000000d210  rbp: 0x00007fff5fbfe850  rsp: 0x00007fff5fbfe850
   r8: 0x0000000000000003   r9: 0x0000000000000000  r10: 0x00007fff88f0597a  r11: 0x0000000000000206
  r12: 0x0000000100115478  r13: 0x00007fff5fbfecd0  r14: 0x00007fff5fbfecd0  r15: 0x00007fff5fbfed20
  rip: 0x00007fff87b59d0b  rfl: 0x0000000000010297  cr2: 0x000000010009b210

Binary Images:
       0x100000000 -        0x10003cfef +ocspd ??? (???)  /usr/sbin/ocspd
    0x7fff5fc00000 -     0x7fff5fc3bdef  dyld 132.1 (???)  /usr/lib/dyld
    0x7fff80853000 -     0x7fff80859ff7 2.3 (2.3)  /System/Library/Frameworks/DiskArbitration.framework/Versions/A/DiskArbitration
    0x7fff80baa000 -     0x7fff80c27fef  libstdc++.6.dylib 7.9.0 (compatibility 7.0.0)  /usr/lib/libstdc++.6.dylib
    0x7fff811af000 -     0x7fff81268fff  libsqlite3.dylib 9.6.0 (compatibility 9.0.0)  /usr/lib/libsqlite3.dylib
    0x7fff812e9000 -     0x7fff8161dfef 861.39 (861.39)  /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CarbonCore.framework/Versions/A/CarbonCore
    0x7fff82092000 -     0x7fff82122fff 1.3.0 (1.3.0)  /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/SearchKit.framework/Versions/A/SearchKit
    0x7fff8229d000 -     0x7fff822c8ff7  libxslt.1.dylib 3.24.0 (compatibility 3.0.0)  /usr/lib/libxslt.1.dylib
    0x7fff82695000 -     0x7fff826d2fff 2.0 (120.1)  /System/Library/Frameworks/LDAP.framework/Versions/A/LDAP
    0x7fff82954000 -     0x7fff82966fe7  libsasl2.2.dylib 3.15.0 (compatibility 3.0.0)  /usr/lib/libsasl2.2.dylib
    0x7fff82ae1000 -     0x7fff82b81fff 362.3 (362.3)  /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/LaunchServices.framework/Versions/A/LaunchServices
    0x7fff82c8f000 -     0x7fff82c9dff7  libkxld.dylib ??? (???)  /usr/lib/system/libkxld.dylib
    0x7fff82d0e000 -     0x7fff82dc4ff7  libobjc.A.dylib 227.0.0 (compatibility 1.0.0)  /usr/lib/libobjc.A.dylib
    0x7fff82dc5000 -     0x7fff82de6fff  libresolv.9.dylib 41.0.0 (compatibility 1.0.0)  /usr/lib/libresolv.9.dylib
    0x7fff8349e000 -     0x7fff834adfff 3.2.2 (3.2.2)  /System/Library/Frameworks/NetFS.framework/Versions/A/NetFS
    0x7fff83a2b000 -     0x7fff83a41fef  libbsm.0.dylib ??? (???)  /usr/lib/libbsm.0.dylib
    0x7fff83a42000 -     0x7fff83bb9fe7 6.6.5 (550.43)  /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
    0x7fff83bbe000 -     0x7fff83c92fe7 454.12.4 (454.12.4)  /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CFNetwork.framework/Versions/A/CFNetwork
    0x7fff83c93000 -     0x7fff83cbbfff 1.1.2 (1.1.2)  /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/DictionaryServices.framework/Versions/A/DictionaryServices
    0x7fff84905000 -     0x7fff849c2fff 359.2 (359.2)  /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/OSServices.framework/Versions/A/OSServices
    0x7fff849d8000 -     0x7fff84a38fe7 2.0 (???)  /System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
    0x7fff850ef000 -     0x7fff8513bfff  libauto.dylib ??? (???)  /usr/lib/libauto.dylib
    0x7fff851a2000 -     0x7fff851ddfff 496.5 (496.5)  /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/AE.framework/Versions/A/AE
    0x7fff8525f000 -     0x7fff85270ff7  libz.1.dylib 1.2.3 (compatibility 1.0.0)  /usr/lib/libz.1.dylib
    0x7fff85b42000 -     0x7fff85b43ff7 1.1 (1)  /System/Library/PrivateFrameworks/TrustEvaluationAgent.framework/Versions/A/TrustEvaluationAgent
    0x7fff87921000 -     0x7fff8796bff7 10.6.3 (507.15)  /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/Metadata.framework/Versions/A/Metadata
    0x7fff87b2b000 -     0x7fff87db4ff7 6.1.2 (55002)  /System/Library/Frameworks/Security.framework/Versions/A/Security
    0x7fff88b46000 -     0x7fff88b4aff7  libmathCommon.A.dylib 315.0.0 (compatibility 1.0.0)  /usr/lib/system/libmathCommon.A.dylib
    0x7fff88f02000 -     0x7fff890c3fef  libSystem.B.dylib 125.2.11 (compatibility 1.0.0)  /usr/lib/libSystem.B.dylib
    0x7fff8921b000 -     0x7fff8933afe7  libcrypto.0.9.8.dylib 0.9.8 (compatibility 0.9.8)  /usr/lib/libcrypto.0.9.8.dylib
    0x7fff8933b000 -     0x7fff89378ff7  libssl.0.9.8.dylib 0.9.8 (compatibility 0.9.8)  /usr/lib/libssl.0.9.8.dylib
    0x7fff89494000 -     0x7fff894d5fff 1.10.8 (1.10.2)  /System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
    0x7fff8953e000 -     0x7fff896fcfff  libicucore.A.dylib 40.0.0 (compatibility 1.0.0)  /usr/lib/libicucore.A.dylib
    0x7fff89b66000 -     0x7fff89b66ff7 44 (44)  /System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
    0x7fff8a545000 -     0x7fff8a65cfef  libxml2.2.dylib 10.3.0 (compatibility 10.0.0)  /usr/lib/libxml2.2.dylib
    0x7fffffe00000 -     0x7fffffe01fff  libSystem.B.dylib ??? (???)  /usr/lib/libSystem.B.dylib

Some of the information you get when googling for the error message is misleading, suggesting ocspd is somehow tied to nVidia graphics drivers. It is in fact the system daemon that handles verification and revocation of SSL certificates using the Online Certificate Status Protocol (OCSP), a vital component of the Internet’s security architecture, as evidenced by the recent Diginotar fiasco. Thus presumably it is used throughout Internet apps like or Chrome, and the regular crashes (at 3 minute intervals or so) would also freeze any apps that make use of cryptography.

The solution was to delete the the temporary certificate revocation list (CRL) databases ocspd maintains:

sudo rm -rf /private/var/db/crls/*
sudo rm -rf /private/var/db/crls/.fl*

A corrupted database is probably responsible for the repeated crashes I observed, and clearing those solved my problem. You may want to make backups of those files instead of just deleting them.

As usual, I disclaim responsibility for any harm this procedure may do to your computer, or induce it to eat your dog, who ate your homework.

So much for the theory that the Mac “just works”. To paraphrase Churchill, it is the worst operating system, with the exception of all others…

Update (2014-04-30):

Well, at least OCSP works on the Mac. You can test it by trying Google Chrome also rejects this, but being Google, they don’t use the standard and implemented their own seriously flawed CRL distribution mechanism  (they manually blacklisted the GRC test site, for instance). Safari on iOS does not block the site at all (a StackOverflow thread suggests verification is only performed for EV certificates, which is are used in only a tiny minority of SSL/TLS sites).

Clearing custom crop aspect ratios in Lightroom

Lightroom’s crop tool allows you to constrain the aspect ratio to a proportion of your choice, e.g. to 4:3, defaulting to the same aspect ratio as the original. The last 5 or so custom crop aspect ratios are saved, but a minor annoyance is you are unable to clear the list.

Python on the Mac and SQLite to the rescue: this simple script will reset them. If you use a non-default name for your Lightroom catalog, you will need to edit it. To run it, quit Lightroom and run the script. It will back up your catalog for you just in case.

Needless to say, I cannot be held liable if this script corrupts your catalog or eats your dog (who ate your homework), use at your own risk.

import sys, os, sqlite3

# edit this to point to your LR3 catalog if you do not use the default location
lrcat = os.path.expanduser('~/Pictures/Lightroom/Lightroom 3 Catalog.lrcat')

os.system('cp -i "%s" "%s.bak"' % (lrcat, lrcat))
db = sqlite3.connect(lrcat)
c = db.cursor()
c.execute("""select value from Adobe_variablesTable
where name='Adobe_customCropAspects'""")
crops = c.fetchone()[0]
print 'aspect ratios:', crops
c.execute("""update Adobe_variablesTable
set value='{}'
where name='Adobe_customCropAspects'""")
print 'Custom crop aspect ratios reset successfully'

A waiter for a server

I had to monitor a long-running process on a Solaris server tonight, but didn’t want to stay glued at a computer monitor. A neat trick:

ssh "pwait 17601"; say "batch done"

You would replace 17601 with the process ID of the job you are waiting for, of course. That way, my Mac connects to the server, waits for the job to complete, then gives me an spoken alert when it is done. I can watch a movie, do chores or whatever during that time. I am sure there are equivalent commands to pwait for Linux.

Hey Apple…

Some improvements you should consider:

  • Sync iPods, iPhones and iPads over WiFi. Cables are so twentieth century. Palm had bluetooth sync working ten years ago, and 802.11n has the same real-world speed as USB. You could then simply extend this to sync the device to the cloud instead of a specific computer.
  • Ditching DVDs to offer an OS reinstall USB flash drive on the new MacBook Airs and Pros is a good idea, but the stick is easy to misplace. How about soldering a read-only USB drive directly onto the motherboard so it can never be lost?
  • When someone enters an address in a Calendar entry on iOS, make it clickable and linked to the Maps app, the way addresses in Contacts are. Copying and pasting them manually is a drag.
  • Stop adding useless frills like “stationery” to, and make the default chronological sort order switchable to “most recent on top”.
  • Add HDMI CEC support to the AppleTV. It would be nice to have a HDTV automatically switch over to the AppleTV’s HDMI input when you try to access it. Speaking of which, it would be nice to have an option to disable the audio out on HDMI, e.g. if you have a decent surround sound system connected to it over Toslink and don’t want the TV’s tinny speakers to kick in.

Dear Parallels

Since you keep hitting me with these spammy popups no matter how many times I click on “Do not show again”, you leave me no choice but to switch to VirtualBox (much better software in any case, and less Windows integration means less chances a virus breaking out of the virtualized Windows ghetto.

Oh, and installing MacFuse without asking permission (unlike VMware Fusion): not cool.

Don’t let the door hit you on the way out.

Will Adobe ever learn?

In a triumph of hope over experience, I recently “upgraded” from Adobe CS3 Design Standard to CS5 Design Standard. I hardly ever use Photoshop any more since I started using Aperture and Lightroom (originally a Macromedia product, no matter what the lame “Adobe Photoshop Lightroom” face-saving branding may try to claim), the main driver for the purchase was actually InDesign CS5 and its ePub functions.

Of course, this is Adobe. Previous versions gratuitously included crud like a full Opera install (an older version, insecure, naturally) just to display a splash screen, or a full MySQL install to power Acrobat search. I never install Acrobat, of course, since that bloated and bug-ridden piece of garbage managed to steal the crown for most insecure software from Internet Explorer, no small feat.

Adobe does not want to confuse users with streamlined and efficient software, so they decided to include the mostly useless Growl on-screen notification program to nag you into registering. Increasing bloat and attack surface for malware is not a good idea, nor is interrupting creative people’s flow with interruptions. Of course, helping clients Get Things Done is a low priority at Adobe, as evidenced by their product choices.

You have to pity the Growl developers, whose reputation will suffer from guilt by association. I dislike interruptions and do not find it useful, but many people do and rave about it. They installed it by choice, not as a sneaky drive-by install for slimy marketing purposes.

Some more annoyances in CS5:

  • The pricing for the suite is more than the sum of its parts: $200 each for Photoshop, InDesign or Illustrator, $700 for Design Standard. I suppose they must think Acrobat and their online tie-ins have a value of $100 (hint: they forgot the negative sign).
  • Of course, they won’t let you upgrade individual component applications.
  • On the plus side, they now have the decency to include Acrobat on a separate CD, so you can discard it immediately and not risk installing it as a side-effect of installing the apps that are actually useful.
  • The icons were designed by the world’s laziest and most creatively bankrupt designer, just as with CS3 and CS4
  • Performance on a high-end 8-core or 12-core Mac is actually slower than on a lower-end configuration, thanks to legacy cruft and incompetence.
  • It is slower to load on my wife’s MacBook Pro. Each successive version of OS X is faster on the same hardware, Microsoft and Adobe deliver software that gets progressively slower.

In other words, unlike Lightroom, CS5 is designed to be endured, not to delight.

Organizing with Delicious Library

Delicious Library is one of the slickest apps on the Mac, and won countless design accolades. Essentially, it is a database for your books, CDs and DVDs (version 2 added gadgets), and it looks glorious on a large monitor like mine. It seems like a novelty for collector-fondlers, and I myself unfairly dismissed it as a toy in 2008, but behind its playful user interface lies a remarkably powerful organizational tool, and the new 2.5 version has made major improvements in stability and performance after 2 years of relative neglect.

Screenshot of Delicious Library

My wife and I are both avid readers—one of our common dreams is to someday have a home with a dedicated room for a library. We are squarely in the demographic for the Bookshelf porn website. Here is a montage of mine alone, not including the books I reluctantly had to consign to storage, or those in my parents’ basement back in France:

My bookshelves

With well over 900 books, I needed a system to manage. At some point I discovered Delicious Library has a writeable location field in its database for every item, and you can create virtual shelves to organize your books. I literally have one DL shelf for each shelf in my bookshelves, one for each box in storage, and one for all the books I keep at work. This way, I can browse shelf by shelf or box by box, or conversely look up the location of a book I need.

Location data for a book

You may think recording the location of each book would be a mind-numbing task, but Delicious Library makes it effortless. It can read ISBN bar codes using the iSight camera included in most Macs, but a better option is to use the Microvision RoV Bluetooth barcode scanner they sell and support. It is quite inexpensive at well over $200, specially compared to the inexpensive Symbol CS1504, but the convenience makes it well worth the price if you have a serious library to wrangle.

If you select a shelf on the right sidebar, then scan a barcode, the book is automatically added to the selected shelf. I filed a feature request with them in 2008 for precisely this, and I do not know when they added it, but it makes a world of difference. The scanner has a memory, so you can zip into an adjoining room, scan all the barcodes on a shelf, zip back and let the scanner pour the shelf’s inventory into DL. It even reads out the titles as they are added. Repeat the process and quite quickly you can compile an inventory of your entire library.

The search function in DL is somewhat primitive, but Smart bookshelves (similar to iTunes’ Smart playlists) help. I have Smart Bookshelves for:

  • Books I have already read
  • Books I have yet to read
  • Books that now have significant resale values on Amazon, to identify candidates for decluttering (you would be surprised to see the markups some art or technical books can fetch once they go out of print, even temporarily, such as my old copy of De Marco’s Peopleware).
  • Books signed by the author
  • Science-Fiction and Fantasy books, using Amazon’s (probably Bookscan’s) categorization
  • Computer books, similarly

Delicious Library tries hard to use cutting-edge functionality in OS X, which is why version 2 only supported Snow Leopard and later. It has a top-notch AppleScript implementation. I am no AppleScript guru, but I relatively easily wrote my own scripts to:

  • Copy the bookshelf name into all the books it contains. This essentially eliminates the data entry, but you have to be careful to make sure a book is not misfiled into two shelves at the same time. The sample script Highlight shelves containing selected media is helpful in this respect.
  • Find the BookMooch entry for books I want to give away.
  • Find the book in the San Francisco Public Library, in case they do not have it and might be interested in a donation.

Thanks to the user interface, browsing books on the Mac is almost as fun as doing so on the bookshelves, and infinitely faster. Of course, this is true of most book catalog programs, including many fine free options available for Windows and Mac, but most others do it with less aplomb.

Incensed at Mozilla

One of the greatest features in the Webkit-based browsers (Apple’s Safari and Google Chrome) is WebSQLdatabase, the ability for a web site to store information in a SQLite database on your browser accessible via JavaScript. This allows web developers to build database-enabled applications that run entirely in the browser, without requiring a server. This is very useful for mobile devices, which in the US enjoy flaky network connectivity at best. One very handsome example is the iPad-optimized Every Time Zone webapp.

SQLite is probably the most important open-source project you have never heard of. It is a simple, streamlined and efficient embedded database. Firefox stores its bookmarks in it. Google distributes its database of phishing sites in that format. Sun’s industrial-strength Solaris operating system stores the list of services it runs on boot in it—if it were to fail, a server would be crippled so that is a pretty strong vote of confidence. Adobe Lightroom and Apple’s Aperture use it to store their database, as do most Mac applications that use the CoreData framework, and many iPhone apps. In other words, it is robust and proven mission-critical software that is widely yet invisibly deployed.

WebSQLdatabase basically makes the power of SQLite available to web developers trying to build apps that work offline, specially on mobile devices. No good deed goes unpunished, and the Mozilla foundation teamed up with unlikely bedfellow Microsoft to torpedo formal adoption of WebSQLdatabase as a web standard, on spurious grounds, and pushed an alternate standard called IndexedDB instead. To quote the Chromium team:

Q: Why this over WebSQLDatabase?

A: Microsoft and Mozilla have made it very clear they will not implement SQL in the browser.  If you want to argue this is silly, talk to them, not me.

IndexedDB is several steps backwards. Instead of using powerful, expressive and mature SQL technology, it uses a verbose JavaScript B-tree API that is a throwback to the 1960s bad old days of hierarchical databases and ISAM, requires a lot more work from the developer, for no good reason. To add injury to insult, Firefox 4’s implementation of IndexedDB is actually built on top of SQLite. The end result will be that web developers will need to build a SQL emulation library on top of IndexedDB to restore the SQLite functionality deliberately crippled by IndexedDB. If there is one constant in software engineering, it is that multiple layers add brittleness and impair performance.

Of course, both Mozilla and Microsoft are irrelevant on mobiles, where WebKit has essentially won the day, so why should this matter? Microsoft has always been a hindrance to the development of the web, since they have to protect the Windows API from competition by increasingly capable webapps, but I cannot understand Mozilla’s attitude, except possibly knee-jerk not-invented-here syndrome and petulance at being upstaged by WebKit. WebSQLdatabase is not perfect—to reach its full potential, it needs and automatic replication and sync facility between the local database and the website’s own database, but it is light years ahead of IndexedDB in terms of power and productivity.

I am so irritated by Mozilla’s attitude that after 10 years of using Mozilla-based browsers, I switched today from Firefox to Chrome as my primary browser. Migrating was surprisingly easy. Key functionality like bookmark keywords, AdBlock, FlashBlock, a developer console, and the ability to whitelist domains for cookies, all have equivalents on Chrome. The main regressions are bookmark tags, and Chrome’s sync options are not yet equivalent to Weave‘s. At some point I will need to roll my own password syncing facility (Chrome stores its passwords in the OS X keychain, which is also used by Safari and Camino).

Ginormous iPod to go

The hard drive in my October 2006 vintage 80GB iPod 5.5G died a few weeks ago.  I wasn’t keen on upgrading to the iPod Classic as:

  1. With a maximum capacity of 160GB, it is still too small to house my entire 220GB music collection
  2. Apple introduced encrypted audio outputs on the dock connector, to force accessory makers to pay royalties, thus making it incompatible with many accessories and forcing you to buy new ones.

I use my iPod mostly in my car. The classic hard drive iPods have one key capability iPhones and iPod Touches lack—the ability to shuffle by album, which is essential when you listen mostly to classical music and where an opus maps to an album.

While investigating repair options, I found out Toshiba now makes a two-platter 240GB (224 GiB) hard drive. The iPod Classic won’t recognize the second platter (a third strike against it) but the 5.5G will. I sent mine to RapidRepair for repair/upgrade and received it back yesterday. The flip side of such an enormous drive is that the sync takes forever: I started it around 10PM yesterday and it is till running, over 9 hours later. They handled the repair very professionally, there are no marks on the casing, and I now have a fully functional 224GB iPod for less than the price of buying a new 160GB iPod Classic. The only feature it is missing is the ability to play 24-bit/96kHz ALAC files like those I made out FLACs purchased from Linn or the B&W Society of Sound.

I can’t understand why Apple does not make this new high-capacity drive available in iPods or the MacBook Air.