IT

Making ScanSnap Receipts usable

For a long time I used a service called Shoeboxed to scan and organize my credit card receipts. Basically stuff your receipts in a US prepaid envelope, drop it in the mail, and they scan, OCR and shred them, as well as analyzing the text to extract the information. Unfortunately, since I moved to the UK the service leaves to be desired, and the price has also gone up over time.

I have a couple of Fujitsu ScanSnap document scanners, a S1500M, which is no longer supported on macOS Mojave (but fortunately is by the third-party app ExactScan), and an iX100 which still is supported, as well as a SV600 which is utterly unsuited to dealing with crumpled receipts. The new, dumbed-down ScanSnap Home app that ships with ScanSnaps has a receipt mode. Since I never use the iX100 to scan documents at home given I have a S1500M (it’s a handheld battery=powered simplex scanner that’s mostly intended for mobile use), I dedicated it and ScanSnap Home to scanning receipts.

The basic functionality of scanning, deskewing, OCR-ing and extracting date, amount, vendor and so on mostly works, but otherwise ScanSnap Receipts is an ergonomic disaster. For starters, it never recognizes the currency correctly and always identifies my transactions as being in dollars rather than pounds. Secondly, it inexplicably lacks the ability to batch-edit receipts, e.g. select the date range for my last trip to France and change all of them from dollars to euros. You need to edit them one by one, which is as incredibly tedious as you can imagine.

After a few weeks of this, I decided to take matters in my own hand. It turns out ScanSnap Home uses Core Data backed by an underlying SQLite database. SQLite is the world’s most widely deployed database (every single Android and iOS smartphone includes it, for starters), but the Core Data object-relational mapper above it does a terrific job of obfuscating it and reducing its performance. Nonetheless, after a little bit of digging, I wrote the following script to automate the most repetitive operations:

  1. Stop all ScanSnap auxiliary processes, as they have the DB opened even if you quit the app
  2. Set the currency for all transactions tagged as Unchecked (ScanSnap Home does this by default) to GBP
  3. Normalize all unchecked Waitrose vendor names to Waitrose
  4. Same for Tesco
  5. Rename M&S to Marks & Spencer
  6. Restart ScanSnap Home

One thing this script cannot do is fix dates. Because date formats are ambiguous (is 11/3/20 March 11 2020, or November 3 2020 or perversely March 20 2011?) and point-of-sale vendors are neither ISO 8601 nor even Y2100 compliant, parsing dates is a minefield.

#!/bin/sh
pkill -9 -f ScanSnap

sqlite3 "$HOME/Library/Application Support/PFU/ScanSnap Home/Managed/ScanSnapHome.sqlite" << EOF

.mode lines

UPDATE zcontent
SET zcurrencysign=(SELECT z_pk FROM zcurrencysign WHERE zvalue='GBP')
WHERE zdoctype=4 AND z_pk IN (
  SELECT z_4contents
  FROM z_4labels
  JOIN zlabel ON z_15labels=zlabel.z_pk
  WHERE zlabel.zname='Unchecked'
);

UPDATE zcontent
SET zvendor=(SELECT z_pk FROM zvendor WHERE zvalue='Waitrose')
WHERE zdoctype=4 AND z_pk IN (
  SELECT z_4contents
  FROM z_4labels
  JOIN zlabel ON z_15labels=zlabel.z_pk
  WHERE zlabel.zname='Unchecked'
) AND zvendor IN (
  SELECT z_pk FROM zvendor
  WHERE zvalue<>'Waitrose' AND zvalue LIKE '%waitrose%'
);

UPDATE zcontent
SET zvendor=(SELECT z_pk FROM zvendor WHERE zvalue='Tesco')
WHERE zdoctype=4 AND z_pk IN (
  SELECT z_4contents
  FROM z_4labels
  JOIN zlabel ON z_15labels=zlabel.z_pk
  WHERE zlabel.zname='Unchecked'
) AND zvendor IN (
  SELECT z_pk FROM zvendor
  WHERE zvalue<>'Tesco' AND zvalue LIKE '%tesco%'
);

UPDATE zcontent
SET zvendor=(SELECT z_pk FROM zvendor WHERE zvalue='Marks & Spencer')
WHERE zdoctype=4 AND z_pk IN (
  SELECT z_4contents
  FROM z_4labels
  JOIN zlabel ON z_15labels=zlabel.z_pk
  WHERE zlabel.zname='Unchecked'
) AND zvendor IN (
  SELECT z_pk FROM zvendor
  WHERE zvalue LIKE '%M&S%'
);

EOF

open /Applications/ScanSnapHomeMain.app

Note that the SQLite database is not used only by expenses (ZCONTENT.ZDOCTYPE=4) but also to store a summary of all documents scanned. Also, the ZCONTENT table has a column ZUNCHECKED that is not what you would expect, it is a constant 1 even if you remove the Unchecked tag from the transaction.

Now, all the usual disclaimers apply, modifying the database directly is not something supported by the app developer, and could have unintended consequences. If you use this script (or more likely modify it for your needs), I disclaim responsibility for any damages or data loss this may cause.

Scanner group test

TL:DR—avoid scanners with Contact Image Sensors if you care at all about color fidelity.

Vermeer it is not

After my abortive trial of the Colortrac SmartLF Scan, I did a comparative test of scanning one of my daughter’s A3-sized drawings on a number of scanners I had handy.

Scanner Sensor Scan
Colortrac SmartLF Scan CIS ScanLF.jpg
Epson Perfection Photo V500 Photo (manually stitched) CCD Epson_V500.jpeg
Epson Perfection V19 (manually stitched) CIS Epson_V19.jpg
Fujitsu ScanSnap S1500M (using a carrier sheet and the built-in stitching) CCD S1500M_carriersheet.jpg
Fujitsu ScanSnap SV600 CCD SV600.jpg
Fuji X-Pro2 with XF 35mm f/1.4 lens, mounted on a Kaiser RS2 XA copy stand with IKEA KVART 3-spot floor lamp (CCT 2800K, a mediocre 82 CRI as measured with my UPRtek CV600) CMOS X-Pro2.jpg

I was shocked by the wide variance in the results, as was my wife. This is most obvious in the orange flower on the right.

Comparison

I scanned a swatch of the orange using a Nix Pro Color Sensor (it’s the orange square in the upper right corner of each scan in the comparison above). When viewed on my freshly calibrated NEC PA302W SpectraView II monitor, the Epson V500 scan is closest, followed by the ScanSnap SV600.

The two scanners using Contact Image Sensor (CIS) technology yielded dismal results. CIS are used in low-end scanners, and they have the benefit of low power usage, which is why the only USB bus-powered scanners available are all CIS models. CIS sensors begat the CMOS sensors used by the vast majority of digital cameras today, superseding CCDs in that application, I would not have expected such a gap in quality.

The digital camera scan was also quite disappointing. I blame the poor quality of the LEDs in the IKEA KVART three-headed lamp I used (pro tip: avoid IKEA LEDs like the plague, they are uniformly horrendous).

I was pleasantly surprised by the excellent performance of the S1500M document scanner. It is meant to be used for scanning sheaves of documents, not artwork, but Fujitsu did not skimp and used a CCD sensor element, and it shows.

Pro tip: a piece of anti-reflective Museum Glass or equivalent can help with curled originals on the ScanSnap SV600. I got mine from scraps at a framing shop. I can’t see a trace of reflections on the scan, unlike on the copy stand.

Update (2018-10-14):

Even more of a pro tip: a Japanese company named Bird Electron makes a series of accessories for the ScanSnap line, including a dust cover for the SV600 and the hilariously Engrish-named PZ-BP600 Book Repressor, essentially a sheet of 3mm anti-reflection coated acrylic with convenient carry handles. They are readily available on eBay from Japanese sellers.

Avery 22807 template for InDesign

The Avery 22807 2-inch circular stickers are a good alternative to Moo, PSPrint et al when you need a small quantity of stickers in a hurry. Unfortunately Avery has not seen it fit to provide usable InDesign templates as they do with some of their other sticker SKUs, only Microsoft Word, which is needless to say inadequate. A search for “Avery 22807 Indesign template” yielded some, but they have issues with missing linked PDF files.

I reverse-engineered the Microsoft template to build one of my own, with dimensions (including the tricky almost-but-not-quite square grid spaced at 5/8″ horizontally but 7/12″ vertically) to simplify “Step and Repeat…”.

I have only tested this with my InDesign CS6, not sure if it will work with older versions.

Avery 22807 2-inch circular labels.indt

Avoiding counterfeit goods on Amazon: mission impossible?

I mentioned previously that I seldom shop for electronics on Amazon.com any more, preferring B&H Photo whenever possible. I now have another reason: avoiding counterfeit goods.

My company boardroom is in an electromagnetic war zone—dozens of competing WiFi access points combined with electronic interference from the US-101 highway just outside make WiFi reception tenuous at best, and unusable more often than not. To work around this, we set up a wired Ethernet switch, and since most of our staff use MacBook Airs, Apple USB Ethernet adapters purchased from Amazon. When I side-graded from my 15″ Retina MacBook Pro to a much more portable 12″ Retina MacBook, I wasn’t able to connect using the dongle, and the name of the device was interspersed with Chinese characters. At first I thought it was an issue with my Satechi USB-C hub, but I experienced the same problems via a genuine Apple USB-C multiport adapter as well.

Eventually I figured out the Ethernet dongles were counterfeit. The packaging, while very similar to Apple’s, was just a tiny bit off, like amateurish margins between the Apple logo and the edges of the card. On the dongles themselves, the side regulatory disclosures sticker was inset, not flush with the body of the adapter.

Counterfeiting is a major problem. By some accounts, one third of all Sandisk memory cards worldwide are counterfeits. In some cases like chargers or batteries, your equipment could be at risk, or even your very life. The counterfeit adapters we purchased from Amazon did not come from Amazon themselves but from a third-party merchant participating in the Amazon marketplace. To Amazon’s credit, we returned them for a prompt, no questions asked refund even though we bought them over six months ago, but it is hard to believe Amazon is unaware of the problem rather than willfully turning a blind eye to it.

My first reaction was to tell our Office Manager to make sure to buy only from Amazon rather than third-party merchants (pro tip: including “amazon” in your Amazon search terms will do that in most cases). Unfortunately, that may not be enough. Amazon has a “fulfilled by Amazon” program for merchants where you ship your goods to them, and they handle warehousing and fulfillment. These “fulfilled by Amazon” items are also more attractive to Prime members. One option Amazon offers is Stickerless, commingled inventory where the items you send are put into a common bin. Amazon still has the ability to trace the provenance of the item through its inventory management, but for purposes of order fulfillment they will be handled just like Amazon’s own stock. Some categories like groceries and beauty products are excluded, but electronics are not.

The implications are huge: even if the vendor is Amazon itself, you cannot be sure that the item is not counterfeit. All the more reason to buy only from trustworthy, single-vendor sites like B&H, even if shipping is a bit slower.

Chrome and AES-256 security: it’s not me, it’s you

This blog now supports the HTTP/2 protocol, courtesy of nginx 1.9.5 (PDF).

In the process, I was stymied by an “ERR_SPDY_INADEQUATE_TRANSPORT_SECURITY” error from Google Chrome. HTTP/2 mandates TLS de facto, if not in the strict letter of the specification, and it also forbids a number of obsolete or weaker SSL/TLS ciphers to only permit ones that are truly secure. After some considerable digging, I found out the issue is Google Chrome on Mac and Android (presumably Windows as well) does not support 256-bit AES in HTTP/2, and my server was set up to only accept 256-bit encryption (only the best will do for my readers!). The error message was misleading: it’s not the server but Chrome’s crypto which is lacking.

It seems the cryptographers at Google feel 128-bit AES in Galois Counter Mode is good enough, and they did not want to be too far apart from Firefox (which does not support it either, and just fails without even the courtesy of an error message). In contrast, Safari on Yosemite supports AES-256-CBC (not ideal, I know, but that’s also what Chrome supports if HTTP/2 is turned off) and AES-256-GCM on El Capitan and iOS 9. Here are the settings your browser uses:

This is disappointing. AES-256-GCM is supported in hardware on most Intel hardware nowadays (all but lowest-end chips have the AES-NI instructions) and in the ARMv8-A architecture supported by most smartphones and mobile devices today, where the extra CPU load would matter most. I wonder how much of this is driven by Google’s fondness for Dan Bernstein’s ChaCha20+Poly1305 algorithms. Excellent as they may be, they are not implemented in hardware on the most common platforms, nor implemented at all in OpenSSL. It is quite disconcerting that my phone has better crypto than my desktop browser.

I ended up resolving the issue by loosening my cipher list from AES256+EECDH to EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH, but Chrome really should catch up and not let itself be hobbled by the increasingly irrelevant Firefox and its hoary NSS crypto.

I probably sound harsher than I intended towards the Google crypto team. The backward compatibility issues they have to deal with, from poorly designed TLS standards to broken web server software, intrusive anti-virus or corporate proxy servers mean a lot of their energy goes into exception cases, rather than implementing the latest and greatest in crypto algorithms.

Update (2017-01-18):

It looks like Chrome silently added AES256-GCM support last year, as it now negotiates the ECDHE-RSA-AES256-GCM-SHA384 cipher on aes256gcm.majid.org.

How the iPad Mini killed my iPhone

The single greatest feature of the iPad is the fact it cannot receive phone calls. Despite being a telecoms engineer by training, I despise phones, and it seems the millennial generation shares my disdain, as it favors less intrusive means of communication like texting.

The iPad is an essential device for me. I am on a 2-year upgrade cycle (at best) for phones, a 5-year cycle for my desktop Mac Pro, and have stopped using laptops altogether, but I will get every single iteration of the iPad. Now, even though my jacket has a pocket sized large enough to hold my full-sized iPad, the weight and bulk means I seldom did so, and kept it in my bag, which I rarely take out with me when going out for lunch. When I saw the iPad Mini and how lightweight it was, I bought one and started carrying it with me all the time.

The Mini is not a replacement for my Retina iPad, as my worsening eyesight makes it a strain for sustained reading, which is why I kept my grandfathered unlimited AT&T data plan on the full-sized iPad and got a limited Verizon plan on the Mini.

No, the device that was displaced is actually my iPhone. The iPad Mini weighs barely twice as much, is thinner, fits in my jacket pocket but has a screen 4 times the size while remaining single-hand-holdable, and is actually usable as a web browsing device or eBook reader, unlike the iPhone’s cramped screen. I don’t believe in the 5-inch phablet form factor, which combines the cramped screen of a phone with the the bulk of a tablet, i.e. the worst of both worlds. I find I never use the iPhone as anything else than a dumb phone any more. I consume less than 60 minutes of voice per month, and if my wife and my startup’s co-founder would let me, I would ditch mobile phones altogether.

Alas I am unable to cut the wireless phone tether, but there is no point in my spending $100 a month on an unlimited data plan for my Verizon iPhone 4, so now that my contract ended, I ported my number over to my old unlocked AT&T iPhone 3GS with a prepaid plan from Airvoice (a MVNO that has the cheapest rates I could find online). At $0.10 a minute without any exorbitant cellco taxes or spurious surcharges, I can expect to spend $6 a month, or 94% savings. That more than covers the $20 a month I pay extra for the iPad Mini’s data plan. The only reason I still use an iPhone instead of switching to a dumbphone is the automatic address book synchronization with my Mac and iOS devices.

If WordPress updates hang on a 64-bit OS

The WordPress instance running this site was no longer able to automatically update plugins (and presumably not the core either) after I upgraded from a 32-bit to a sparkling fresh 64-bit PHP install at Joyent. It would start the update, and show a spinning logo and then just hang.

After much debugging, I found out the problem is that the class-pclzip.php that is responsible for unzipping was failing silently with the message:

Downloading update from http://downloads.wordpress.org/plugin/yet-another-related-posts-plugin.3.5.2.zip

Unpacking the update…

Abort class-pclzip.php : Missing zlib extensions

This isn’t terribly helpful, but digging in, it turns out that class depends on the PHP zlib module, and on 64-bit operating systems (more precisely, operating systems with 64-bit large file support enabled), zlib.h #defines gzopen to be gzopen64. PHP does not protect itself adequately and thus the PHP function gzopen gets renamed gzopen64 as well, this throwing class-pclzip.php for a loop, along with a number of other systems like PEAR.

Fixing this requires recompiling PHP. Ubuntu Karmic includes a work-around, but I run Solaris and build from source, so I contributed a patch filed under bug #53829.

Automattic should probably patch class-pclzip.php to deal with gzopen/gzopen64 as there are a great many broken PHP installs out there (the PHP bug has been open for over a year and a half without what I would consider an acceptable solution), and it is surprisingly difficult to find a solution online. I guess a great many WordPress installs are still 32-bit, which is kind of sad.

Scanning your iTunes library for DRM-infested books

Tor, the leading publisher for Science Fiction and Fantasy books, announced they would be doing away with DRM in their eBooks. The product pages for their books on iBooks now mention “At the publisher’s request, this title is being sold without Digital Rights Management software (DRM) applied”. I figured it would be a good idea to uncripple the many Tor eBooks I have in my collection.

I wrote a quick little Python script to scan my growing iBooks library for books that could be updated. The procedure is to delete the book from both iTunes and iPads, then download it anew (restarting iTunes is also needed after deleting). Apple keeps track of your purchases and will not charge you again.

#!/usr/bin/env python
import sys, os.path, glob, zipfile, platform, xml.etree.ElementTree

# publishers who have forsaken DRM
good = ['Tom Doherty']

if platform.mac_ver()[0] > '10.8':
  bookdir = os.path.expanduser('~/Library/Containers/com.apple.BKAgentService/Data/Documents/iBooks')
else:
  bookdir = os.path.expanduser('~/Music/iTunes/iTunes Music/Books')

os.chdir(bookdir)

ok =  '\033[1;32mDRM-free    \033[0m'
bad = '\033[1;31mDRM-infested\033[0m'

count = 0
salvageable = 0

def extract(meta):
  creator = ''
  status = ''
  pub = ''
  et = xml.etree.ElementTree.fromstring(meta)
  try:
    creator = et.findall('*{http://purl.org/dc/elements/1.1/}creator')
    creator = creator[0].text
    title = et.findall('*{http://purl.org/dc/elements/1.1/}title')
    title = title[0].text
  except:
    assert '!DOCTYPE plist' in meta
    next_tag = None
    for e in et[0].iter():
      if e.tag == 'key' and e.text in ('artistName', 'itemName'):
        next_tag = e.text
        continue
      if next_tag == 'artistName':
        creator = e.text
        next_tag = None
        continue
      elif next_tag == 'itemName':
        title = e.text
        next_tag = None
        continue
  pub = [x for x in good if x in meta]
  return creator, title, pub

def find_meta(file_list, opener):
  for m in file_list:
    if m.endswith('.opf') or m == 'iTunesMetadata.plist':
      meta = opener(m).read()
      return extract(meta)
  
for fn in glob.glob('*/*.epub'):
  status = ok
  suffix = ''
  if os.path.isdir(fn):
    suffix = '(directory)'
    if os.path.exists(fn + '/META-INF/encryption.xml'):
      status = bad
      count += 1
    meta = find_meta(os.listdir(fn), lambda x: open(fn + '/' + x))
  else:
    z = zipfile.ZipFile(fn)
    try:
      i = z.getinfo('META-INF/encryption.xml')
      status = bad
    except KeyError:
      pass
    meta = find_meta(z.namelist(), z.open)
    z.close()
  creator, title, pub = meta
  print status, fn, suffix
  print '\t', creator
  print '\t', title
  if status == bad and pub:
    print '\t\033[1;32mThis is published by', pub[0],
    print 'and could be re-downloaded DRM-free\033[0m'
    salvageable += 1

print count, 'books are DRM-infested'
print salvageable, 'could be cured'

Unfortunately, it seems like the DRM-stripping is still work in progress. Out of the Wheel of Time series, for instance, only the first one is now DRM-free on the iBooks store.

undr ~>drmbooks.py
DRM-free     Books/0083D0AEC37E08453347DD12B1C6F980.epub 
	Greg Bear
	Blood Music
DRM-free     Books/09178837756A4DFF8347EC377345A37B.epub 
	Heinz Wittenbrink
	RSS and Atom
DRM-free     Books/0AD752E995042C7E12F11917AB58C6B8.epub 
	Wes McKinney
	Python for Data Analysis
DRM-free     Books/14BDC66A99E878EC232FFAFA73B341EF.epub 
	Fritz Leiber
	Swords and Deviltry-Fafhrd and the Gray Mouser-Book1
DRM-free     Books/15A1D7FE9B7D815C6FBE1A9A77D7143E.epub 
	Glen Cook
	A Fortress in Shadow
DRM-free     Books/1793F9DE1319B96FDE7E36EB8A1BC961.epub 
	Scalzi, John
	Old Man's War
DRM-free     Books/1D08BE221E8BC8F2A371EFEDE55029AC.epub 
	Ben Fry
	Visualizing Data
DRM-free     Books/24D6EC36CDEA0C1E8612CC61A89EA098.epub 
	None
	Node Cookbook
DRM-free     Books/29DA285F0051C431BD8BA3D1AEC5EAA6.epub 
	Fritz Leiber
	The Swords of Lankhmar: Fafhrd and the Gray Mouser-Book 5
DRM-free     Books/2E88CD68DFD8408CD0E7C0ACB1E78714.epub 
	Glen Cook
	A Cruel Wind: A Chronicle of the Dread Empire
DRM-free     Books/32996A9995040064818BAE4DFB66E92F.epub 
	Kelly Link
	Magic for Beginners
DRM-free     Books/34D3CD13D47E5FEBC6DCF7EF011113BD.epub 
	David Drake
	Lord of the Isles
DRM-infested Books/357298432.epub 
	Iain M. Banks
	The Player of Games
DRM-infested Books/357311036.epub 
	Iain M. Banks
	Use of Weapons
DRM-infested Books/357377857.epub 
	Iain M. Banks
	Against a Dark Background
DRM-infested Books/357396585.epub 
	Brent Weeks
	Night Angel: The Complete Trilogy
DRM-infested Books/357657026.epub 
	Iain M. Banks
	Transition
DRM-infested Books/357658374.epub 
	Po Bronson
	NurtureShock: New Thinking About Children
DRM-infested Books/357662058.epub 
	Iain M. Banks
	Consider Phlebas
DRM-infested Books/357669769.epub 
	Iain M. Banks
	Matter
DRM-infested Books/357914731.epub 
	Herbert, Frank
	Dune Messiah
DRM-infested Books/357918110.epub 
	Dalrymple, William
	City of Djinns
DRM-infested Books/357923567.epub 
	Patrick Rothfuss
	The Name of the Wind
DRM-infested Books/357929995.epub 
	Herbert, Frank
	Dune (40th Anniversary Edition)
DRM-infested Books/357969577.epub 
	Herbert, Frank
	God Emperor of Dune
DRM-infested Books/357987322.epub 
	Herbert, Frank
	Children of Dune
DRM-infested Books/357994537.epub 
	Herbert, Frank
	Heretics of Dune
DRM-infested Books/357994652.epub 
	William Dalrymple
	White Mughals: Love and Betrayal in Eighteenth-Century India
DRM-infested Books/357996119.epub 
	Stross, Charles
	Wireless
DRM-infested Books/360601506.epub 
	Ursula K. Le Guin
	The Dispossessed
DRM-infested Books/360609519.epub 
	Greg Egan
	Schild's Ladder
DRM-infested Books/360627712.epub 
	Raymond E. Feist
	Rides a Dread Legion
DRM-infested Books/360627930.epub 
	Neal Stephenson
	Anathem
DRM-infested Books/360628773.epub 
	Raymond E. Feist
	At the Gates of Darkness
DRM-infested Books/360641088.epub 
	Mihaly Csikszentmihalyi
	Flow
DRM-free     Books/361491495.epub 
	Basil Hall Chamberlain
	Aino Folk-Tales
DRM-free     Books/361494664.epub 
	Poul William Anderson
	Industrial Revolution
DRM-free     Books/361523763.epub 
	Lafcadio Hearn
	The Romance of the Milky Way / And Other Studies & Stories
DRM-free     Books/361527545.epub 
	Saki
	When William Came
DRM-free     Books/361539032.epub 
	Saki
	The Chronicles of Clovis
DRM-free     Books/361557387.epub 
	Saki
	Reginald in Russia and other sketches
DRM-free     Books/361557834.epub 
	Sir Arthur Conan Doyle
	The Adventure of the Dying Detective
DRM-free     Books/361559391.epub 
	Sir Arthur Conan Doyle
	The Valley of Fear
DRM-free     Books/361560694.epub 
	Lafcadio Hearn
	Chita: a Memory of Last Island
DRM-free     Books/361561399.epub 
	Confucius
	The Analects of Confucius (from the Chinese Classics)
DRM-free     Books/361562678.epub 
	Saki
	The Toys of Peace, and other papers
DRM-free     Books/361562764.epub 
	Sir Arthur Conan Doyle
	The Memoirs of Sherlock Holmes
DRM-free     Books/361564075.epub 
	Poul William Anderson
	The Burning Bridge
DRM-free     Books/361564898.epub 
	Henry David Thoreau
	Walden
DRM-free     Books/361565201.epub 
	Saki
	Beasts and Super-Beasts
DRM-free     Books/361565806.epub 
	Isaac Asimov
	Youth
DRM-free     Books/361572327.epub 
	Lafcadio Hearn
	Kokoro / Japanese Inner Life Hints
DRM-free     Books/361573126.epub 
	Sir Arthur Conan Doyle
	Through the Magic Door
DRM-free     Books/361575882.epub 
	Sir Arthur Conan Doyle
	Tales of Terror and Mystery
DRM-free     Books/361578744.epub 
	Lafcadio Hearn
	In Ghostly Japan
DRM-free     Books/361588265.epub 
	Lafcadio Hearn
	Books and Habits from the Lectures of Lafcadio Hearn
DRM-free     Books/361673695.epub 
	E. C. Babbitt
	More Jataka Tales
DRM-free     Books/361686559.epub 
	Poul William Anderson
	Security
DRM-free     Books/361713863.epub 
	Sir Arthur Conan Doyle
	The Return of Sherlock Holmes
DRM-free     Books/361721797.epub 
	Sir Arthur Conan Doyle
	The Adventure of the Cardboard Box
DRM-free     Books/361725352.epub 
	Saki
	The Unbearable Bassington
DRM-free     Books/361725959.epub 
	Sir Arthur Conan Doyle
	The Adventure of Wisteria Lodge
DRM-free     Books/361726975.epub 
	Saki
	Reginald
DRM-free     Books/361727237.epub 
	Sir Arthur Conan Doyle
	The Adventure of the Red Circle
DRM-free     Books/361732286.epub 
	Poul William Anderson
	The Valor of Cappen Varra
DRM-free     Books/361736043.epub 
	Lafcadio Hearn
	Kwaidan: Stories and Studies of Strange Things
DRM-free     Books/361741563.epub 
	Lafcadio Hearn
	Japan: an Attempt at Interpretation
DRM-free     Books/361743007.epub 
	Ambrose Bierce
	The Devil's Dictionary
DRM-free     Books/361743953.epub 
	Sir Arthur Conan Doyle
	The Adventure of the Devil's Foot
DRM-free     Books/361744178.epub 
	Sir Arthur Conan Doyle
	His Last Bow
DRM-free     Books/361745602.epub 
	Poul William Anderson
	The Sensitive Man
DRM-infested Books/362435686.epub 
	Ansary, Tamim
	Destiny Disrupted
DRM-infested Books/366773380.epub 
	Esslemont, Ian C. C.
	Return of the Crimson Guard
	This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/373338999.epub 
	Jordan, Robert
	The Path of Daggers
	This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-free     Books/375554215.epub 
	Orson Scott Card
	The Lost Gate
DRM-infested Books/376217648.epub 
	Steven Erikson
	Reaper’s Gale
	This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/376227359.epub 
	Jordan, Robert
	Winter's Heart
	This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/376227401.epub 
	Jordan, Robert
	Crossroads of Twilight
	This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/376227406.epub 
	Jordan, Robert
	Knife of Dreams
	This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/376227409.epub 
	Sanderson, Brandon
	The Gathering Storm
	This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/376227423.epub 
	Jordan, Robert
	New Spring
	This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-free     Books/376231110.epub 
	Cook, Glen
	Surrender to the Will of the Night
DRM-infested Books/376231528.epub 
	Robert Jordan and Brandon Sanderson
	Towers of Midnight
	This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/378317076.epub 
	Jordan, Robert
	A Crown of Swords
	This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/378317808.epub 
	Robert Jordan
	Lord of Chaos
	This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-free     Books/379451459.epub 
	Le Guin, Ursula K.
	Word for World is Forest, The
DRM-free     Books/37EC6E895E6BD70BEB48D9F1553D608E.epub 
	Eben Hewitt
	Cassandra: The Definitive Guide
DRM-free     Books/380490608.epub 
	Jordan, Robert
	The Eye of the World
DRM-free     Books/380494444.epub 
	Asimov, Isaac
	The End of Eternity
DRM-infested Books/381497257.epub 
	Harold McGee
	On Food and Cooking, The Science and Lore of the Kitchen
DRM-infested Books/381622032.epub 
	IAIN M. BANKS
	Look to Windward
DRM-infested Books/381683084.epub 
	Ursula K. Le Guin
	Tehanu
DRM-infested Books/381935940.epub 
	Richard Adams
	Watership Down
DRM-infested Books/382674388.epub 
	Steven Erikson
	Dust of Dreams
	This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/383912791.epub 
	Steven Erikson
	Bauchelain and Korbal Broach
	This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/385975662.epub 
	Jordan, Robert
	The Dragon Reborn
	This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/385981104.epub 
	Steven Erikson
	The Bonehunters
	This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/385981116.epub 
	Steven Erikson
	Midnight Tides
DRM-free     Books/385982966.epub 
	Brust, Steven
	To Reign in Hell
DRM-infested Books/385987858.epub 
	Steven Erikson
	Gardens of the Moon
	This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/385989170.epub 
	Steven Erikson
	House of Chains
DRM-infested Books/385992628.epub 
	Jordan, Robert
	The Fires of Heaven
	This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/385992927.epub 
	Steven Erikson
	Toll the Hounds
	This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/385992930.epub 
	Jordan, Robert
	The Great Hunt
	This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/385998417.epub 
	Jordan, Robert
	The Shadow Rising
	This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/386016540.epub 
	Esslemont, Ian C. C.
	Night of Knives
	This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/388403394.epub 
	Steven Erikson
	The Crippled God
DRM-infested Books/389191300.epub 
	Iain M. Banks
	Surface Detail
DRM-infested Books/390877859.epub 
	Loewen, James W.
	Lies My Teacher Told Me
DRM-infested Books/393310992.epub 
	Erikson, Steven
	Memories of Ice
DRM-infested Books/394745271.epub 
	Steven Erikson
	Deadhouse Gates
	This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-free     Books/394745833.epub 
	Walton, Jo
	Among Others
DRM-free     Books/395536306.epub 
	Sir Arthur Conan Doyle
	The Adventures of Sherlock Holmes
DRM-free     Books/395537209.epub 
	Sir Arthur Conan Doyle
	The Sign of the Four
DRM-free     Books/395539542.epub 
	Sir Arthur Conan Doyle
	A Study in Scarlet
DRM-free     Books/395540660.epub 
	Sir Arthur Conan Doyle
	The Hound of the Baskervilles
DRM-free     Books/395686685.epub 
	Dante Alighieri
	Divine Comedy, Longfellow's Translation, Complete
DRM-free     Books/395688318.epub 
	Edgar Rice Burroughs
	A Princess of Mars
DRM-free     Books/395688375.epub 
	Lafcadio Hearn
	Glimpses of an Unfamiliar Japan / First Series
DRM-infested Books/395926792.epub 
	Fukuyama, Francis
	Origins of Political Order
DRM-infested Books/396269736.epub 
	Herbert, Frank
	Chapterhouse: Dune
DRM-infested Books/398283114.epub 
	Rothfuss, Patrick
	The Wise Man's Fear
DRM-free     Books/3A5FBC58E821CFDF15C8C4E85657481E.epub 
	Jon Hicks
	The Icon Handbook
DRM-free     Books/410943153.epub 
	Edwin A. Abbott (A Square)
	Flatland: A Romance of Many Dimensions
DRM-free     Books/413463878.epub 
	Brust, Steven
	Tiassa
DRM-free     Books/418293515.epub 
	Heinlein, Robert A.
	Glory Road
DRM-infested Books/419950945.epub 
	Isaac Asimov
	Foundation
DRM-infested Books/419950970.epub 
	Isaac Asimov
	Foundation and Empire
DRM-infested Books/419950976.epub 
	Isaac Asimov
	Second Foundation
DRM-infested Books/419968238.epub 
	Scott Lynch
	The lies of Locke Lamora
DRM-infested Books/419968784.epub 
	Scott Lynch
	Red Seas Under Red Skies
DRM-infested Books/420037362.epub 
	Kim Stanley Robinson
	The Years of Rice and Salt
DRM-infested Books/420281728.epub 
	Richard Wiseman
	59 Seconds: Think a Little, Change a Lot
DRM-infested Books/420445771.epub 
	Isaac Asimov
	Foundation’s Edge
DRM-infested Books/420446058.epub 
	Isaac Asimov
	Foundation and Earth
DRM-infested Books/420725428.epub 
	Mike Resnick
	Kirinyaga: A Fable of Utopia
DRM-infested Books/421025353.epub 
	William Dalrymple
	The Last Mughal
DRM-free     Books/421124117.epub 
	Brust, Steven
	The Desecrator
DRM-infested Books/422530144.epub 
	Max Barry
	Machine Man
DRM-free     Books/422718511.epub 
	Apple Inc.
	Mac Integration Basics
DRM-free     Books/426914658.epub 
	Brust, Steven
	Five Hundred Years After
DRM-free     Books/428235697.epub 
	Vinge, Vernor
	A Fire Upon The Deep
DRM-infested Books/429173089.epub 
	Ursula K. Le Guin
	The Other Wind
DRM-infested Books/429173713.epub 
	Ursula K. Le Guin
	Tales from Earthsea
DRM-infested Books/429699133.epub 
	Rajaniemi, Hannu
	The Quantum Thief
	This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/431617578.epub 
	Walter Isaacson
	Steve Jobs
DRM-infested Books/432519291.epub 
	Susan Weinschenk
	100 Things: Every Designer Needs to Know About People
DRM-infested Books/434509014.epub 
	Stross, Charles
	Rule 34
DRM-free     Books/434522188.epub 
	Larry Niven, Jerry Pournelle
	The Mote In God's Eye
DRM-free     Books/434811509.epub 
	Asher, Neal
	Cowl
DRM-infested Books/436646026.epub 
	Neal Stephenson
	Reamde
DRM-infested Books/436691174.epub 
	Julia Child
	Mastering the Art of French Cooking
DRM-infested Books/443149884.epub 
	Daniel Kahneman
	Thinking, Fast and Slow
DRM-infested Books/446155927.epub 
	Esslemont, Ian C. C.
	Stonewielder
	This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-free     Books/447591195.epub 
	Asher, Neal
	The Skinner
DRM-infested Books/454252718.epub 
	William B. Norton
	The Internet Peering Playbook: Connecting to the Core of the Internet
DRM-infested Books/455525627.epub 
	Amar Chitra Katha
	Birbal The Genius
DRM-infested Books/458461362.epub 
	Pamela Druckerman
	Bringing Up Bebe
DRM-free     Books/45B90418E467D479DCDDF23B932C648C.epub 
	Douglas Crockford
	JavaScript: The Good Parts
DRM-infested Books/460822066.epub (directory)
	Scott Lynch
	The Republic of Thieves
DRM-free     Books/465679A557523FDB836005CF4BB9380E.epub 
	Scott Berkun
	Mindfire
DRM-infested Books/479594044.epub 
	Saladin Ahmed
	Throne of the Crescent Moon
DRM-infested Books/479717436.epub 
	David Crist
	The Twilight War: The Secret History of America’s Thirty-Year Conflict with Iran
DRM-infested Books/479771801.epub 
	William Dalrymple
	In Xanadu
DRM-infested Books/489957500.epub 
	Bruce Schneier
	Liars and Outliers
DRM-infested Books/491186678.epub 
	James Blish
	Cities in Flight
DRM-infested Books/491668459.epub 
	Neal Asher
	Shadow of the Scorpion
DRM-infested Books/491669284.epub 
	Glen Cook
	A Matter of Time
DRM-infested Books/491669288.epub 
	Glen Cook
	Darkwar
DRM-free     Books/492199230.epub 
	Frederik Pohl
	The Tunnel Under the World
DRM-free     Books/492199569.epub 
	Frederik Pohl
	The Knights of Arthur
DRM-infested Books/494939678.epub 
	Esslemont, Ian C. C.
	Orb Sceptre Throne
	This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/498634992.epub 
	Charles Stross
	The Apocalypse Codex
DRM-infested Books/499392787.epub 
	Daniel Suarez
	Kill Decision
DRM-free     Books/4AE7DDA54BEEFCD65157927546B18063.epub 
	Roberto Ierusalimschy
	Programming in Lua 2ed
DRM-free     Books/4DCFF682728B765A1CE221F3D7C21536.epub 
	Glen Cook
	Starfishers Volume 3: Stars' End
DRM-free     Books/501278407.epub 
	Colette
	Chéri
DRM-free     Books/501758197.epub 
	David Brin
	Existence
DRM-infested Books/501758516.epub 
	Scalzi, John
	Redshirts
	This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/503019669.epub 
	J.R.R. Tolkien
	The Lord of the Rings
DRM-infested Books/503153300.epub 
	J.R.R. Tolkien
	Tales from the Perilous Realm
DRM-infested Books/503154129.epub 
	J. R. R. Tolkien and Christopher Tolkien
	The Book of Lost Tales, Part One
DRM-infested Books/503154991.epub 
	J. R. R. Tolkien and Christopher Tolkien
	The Book of Lost Tales, Part Two
DRM-infested Books/503155327.epub 
	J.R.R. Tolkien
	The Children of Húrin
DRM-infested Books/503163148.epub 
	J.R.R. Tolkien
	The Hobbit Deluxe
DRM-infested Books/503164678.epub 
	J.R.R. Tolkien
	The Silmarillion
DRM-infested Books/503167303.epub 
	J.R.R. Tolkien
	Unfinished Tales of Númenor and Middle-earth
DRM-infested Books/504209078.epub 
	Isaac Asimov
	Prelude to Foundation
DRM-infested Books/504371982.epub 
	Iain M. Banks
	The Hydrogen Sonata
DRM-free     Books/511060740.epub 
	Frederik Pohl
	The Hated
DRM-free     Books/511143617.epub 
	Frederik Pohl
	The Day of the Boomer Dukes
DRM-free     Books/511252896.epub 
	Frederik Pohl
	Pythias
DRM-free     Books/513357605.epub 
	Hannu Rajaniemi
	The Fractal Prince
DRM-free     Books/513868CF0AA46D293EE27F74BC399760.epub 
	Jerry Pournelle
	West of Honor
DRM-infested Books/520233773.epub 
	Nate Silver
	The Signal and the Noise
DRM-free     Books/520897403.epub 
	Hugh Howey
	Wool Omnibus
DRM-free     Books/525170910.epub 
	Cory Doctorow and Charles Stross
	The Rapture of the Nerds
DRM-infested Books/526136048.epub (directory)
	Hetty van de Rijt & Frans Plooij
	The Wonder Weeks
DRM-infested Books/529020127.epub 
	Neal Asher
	The Departure
DRM-infested Books/529424632.epub 
	Guy Gavriel Kay
	The Lions of Al-Rassan
DRM-infested Books/536312376.epub 
	Glen Cook
	Garrett for Hire
DRM-free     Books/537023027.epub 
	Erikson, Steven
	Forge of Darkness
DRM-infested Books/541673159.epub 
	Ursula K. Le Guin
	The Tombs of Atuan
DRM-infested Books/541673162.epub 
	Ursula K. Le Guin
	The Farthest Shore
DRM-infested Books/546126326.epub 
	Jan Morris
	HAV
DRM-infested Books/551241606.epub 
	Ursula K. Le Guin
	A Wizard of Earthsea
DRM-infested Books/551567785.epub 
	Murray R. Spiegel, PhD
	Schaum's Outline Mathematical Handbook of Formulas and Tables, Fourth Edition
DRM-infested Books/551747038.epub 
	Chris Hedges and Joe Sacco
	Days of Destruction Days of Revolt v2b
DRM-infested Books/552144691.epub 
	Tamim Ansary
	Games without Rules
DRM-free     Books/553878102.epub 
	Charles Stross
	A Tall Tail
DRM-infested Books/563408849.epub 
	Iain Banks
	Stonemouth
DRM-infested Books/568731449.epub 
	William Dalrymple
	Return of a King: The Battle for Afghanistan, 1839-42
DRM-infested Books/569232538.epub 
	Neil Gaiman
	The Ocean at the End of the Lane
DRM-free     Books/571678152.epub 
	Glen Cook
	The Return of the Black Company
DRM-free     Books/571678945.epub 
	Glen Cook
	The Many Deaths of the Black Company
DRM-free     Books/573656304.epub 
	Glen Cook
	Chronicles of the Black Company
DRM-free     Books/573656441.epub 
	Glen Cook
	The Books of the South
DRM-free     Books/576233114.epub 
	Jack Vance
	Demon Princes
DRM-infested Books/578851675.epub 
	Zilpha Keatley Snyder
	Below The Root
DRM-infested Books/580642602.epub 
	Max Barry
	Lexicon
DRM-infested Books/588794444.epub 
	Charles Stross
	Neptune's Brood
DRM-free     Books/5CCB889F91585202637A3C5FBFD8409F.epub 
	Glen Cook
	Starfishers-The Starfishers Trilogy Volume II
DRM-infested Books/600938002.epub 
	Gardner Dozois
	The Year’s Best Science Fiction: Thirtieth Annual Collection
DRM-free     Books/606232503.epub 
	Ian C. Esslemont
	Blood and Bone
DRM-free     Books/610920977.epub 
	Robert Jordan and Brandon Sanderson
	A Memory of Light
DRM-free     Books/6122FF30560866BD75257E6CCC264371.epub 
	Fritz Leiber
	Swords and Ice Magic-Fafhrd and the Gray Mouser-Book 6
DRM-infested Books/619483561.epub 
	Raymond E. Feist
	Magician's End
DRM-free     Books/622837311.epub 
	Le Comte De  Lautréamont
	Les chants de Maldoror
DRM-free     Books/62497334A4189B1D00E9BDFD95724E2E.epub 
	Fritz Leiber
	The Knight and Knave of Swords-Fafhrd and the Gray Mouser-Book 7
DRM-infested Books/645571245.epub 
	Iain Banks
	The Quarry
DRM-free     Books/647688922.epub 
	Steven Brust and Skyler White
	The Incrementalists
DRM-infested Books/651331715.epub 
	Susan Crawford
	Captive Audience
DRM-infested Books/654456347.epub 
	Iain Banks
	The Wasp Factory
DRM-infested Books/662310218.epub (directory)
	Various Authors
	Star Wars: Empire Volume 3 – The Imperial Perspective
DRM-infested Books/662310219.epub 
	Paul Gulacy
	Star Wars: Crimson Empire
DRM-infested Books/664297397.epub (directory)
	Various Authors
	Star Wars: Empire, Vol. 4: The Heart of the Rebellion
DRM-infested Books/664343525.epub (directory)
	Paul Chadwick, Doug Wheatley & Tomás Giorello
	Star Wars: Empire, Vol. 2: Darklighter
DRM-infested Books/664894567.epub 
	John Ostrander
	Star Wars: Dawn of the Jedi Volume 1—Force Storm
DRM-infested Books/664910993.epub (directory)
	Scott Allie, Ryan Benjamin & Brian Horton
	Star Wars: Empire Vol. 1
DRM-free     Books/666772E4C21E2017E71C10F1990840BC.epub 
	Pieter Hintjens
	ZeroMQ - Connecting your Code
DRM-infested Books/674224604.epub (directory)
	Various Authors
	Star Wars: Empire, Vol. 5: Allies and Adversaries
DRM-infested Books/674226603.epub (directory)
	Thomas Andrews, Scott Allie & Various Authors
	Star Wars: Empire, Vol. 6: In the Shadows of Their Fathers
DRM-free     Books/697938901.epub 
	Charles Stross
	Equoid: A Laundry Novella
DRM-free     Books/6BBD61A64A46FFFF4AE7C5FCEB9CFCEE.epub 
	David Drake
	Balefires
DRM-free     Books/71C79A253586282ABE73C2975237EB08.epub 
	Glen Cook
	Sung in Blood
DRM-free     Books/73BC5DD0E51611BDC359CBAB48CC203F.epub (directory)
	Crane, Stephen
	The Red Badge of Courage
DRM-free     Books/7A0D1AEE343638A8A3769DABB90CD4CB.epub 
	Clay A. Johnson
	The Information Diet
DRM-free     Books/7B46914B55481714DED3D711288978FA.epub 
	Glen Cook
	Shadowline-The Starfishers Trilogy I
DRM-free     Books/809DCC75FE60E749458CB27636EE6777.epub 
	Mercedes Lackey
	The Secret World Chronicle
DRM-free     Books/8350F5DFECCD6AA88714400FDF4F6831.epub 
	François de La Rochefoucauld
	Réflexions ou Sentences et Maximes Morales
DRM-free     Books/853BD1C65277D276BA09E04EBFEB73EF.epub 
	None
	PostgreSQL 9 Administration Cookbook
DRM-free     Books/85907063B2351E91C6E7F5052C090BFD.epub 
	None
	PostgreSQL 9.0 High Performance
DRM-free     Books/8663E7BF735C06318FF450532A67F1C2.epub 
	Glen Cook
	Passage at Arms
DRM-free     Books/8690D69995483C5D2DF6AD38BE53C1D7.epub 
	Paolo Bacigalupi
	The Windup Girl - Second Electronic Edition
DRM-free     Books/8993920EF1C0E8FE8CB47A20BD955F53.epub 
	Fritz Leiber
	Swords Against Death-Fafhrd and Gray Mouser-Book 2
DRM-free     Books/8DC6ED0A99161EBB516E370A60ED1121.epub 
	Jonathan Zdziarski
	Hacking and Securing iOS Applications
DRM-free     Books/8DD0D22CD05B0B8D52DF9DB93FC8616B.epub 
	Fritz Leiber
	Swords in the Mist-Fafhrd And the Gray Mouser-Book 3
DRM-free     Books/912CB8B110736684549EAC4FC36665AB.epub 
	Neil Gaiman and Dave McKean
	Signal to Noise
DRM-free     Books/93061FDFD6EEC01FD8CE4295A049C97C.epub 
	Cory Doctorow
	Homeland
DRM-free     Books/9592D5001632B94D1FEFC98B3A40E049.epub 
	Kelly Link
	Stranger Things Happen
DRM-free     Books/990CA5799084407151488A7C563DF269.epub 
	Tom Hughes-Croucher
	Node: Up and Running
DRM-free     Books/9CCCA9E217602948B84BE9A7A21C2753.epub 
	Mike Resnick
	Birthright: The Book of Man
DRM-free     Books/9DA2C8D7E941C37FA83192E5849AA850.epub 
	Q. Ethan McCallum
	Parallel R
DRM-free     Books/A7DB557515983DB67F74E2BE351FC319.epub 
	Glen Cook
	Reap the East Wind
DRM-free     Books/B18AB85267B815E540C7E370F8D97726.epub 
	Lars George
	HBase: The Definitive Guide
DRM-free     Books/B1A586D24AE5F2450010F8664F8E059D.epub 
	Cory Doctorow
	Pirate Cinema
DRM-free     Books/B562B5D74F8677C946B0A6F81EB34F1B.epub 
	Gotthold Ephraim Lessing
	Nathan the Wise; a dramatic poem in five acts
DRM-free     Books/BE42D426242836AA171539B7415732E6.epub 
	Glen Cook
	The Swordbearer
DRM-free     Books/BE80AD93781746E45CF95607A7BEE687.epub 
	Charlie Stross
	Bit Rot
DRM-free     Books/C39FBC59673713A57D404D62BE85C4DC.epub 
	Lauren Beukes
	Zoo City
DRM-free     Books/C81CBBD470EFDBC62359BDAD12FDF551.epub 
	Thomas Hobbes
	Leviathan
DRM-free     Books/D17DC1A83BD90AFB24055A01831BFAB4.epub 
	Glen Cook
	The Dragon Never Sleeps
DRM-free     Books/DA8FBAB386EC503A0EF21E481D214521.epub 
	David Flanagan
	JavaScript: The Definitive Guide
DRM-free     Books/DBAD0DE7A4305B90F6AC33C673FE68E8.epub 
	Glen Cook
	A Path to Coldness of Heart
DRM-free     Books/DBCCB54B140D1158250B24B7E3E81B63.epub 
	Scott Chacon
	Pro Git
DRM-free     Books/DF6C56F9A8719294B13BA91BED5E1667.epub 
	Mike Resnick
	Ivory
DRM-free     Books/E8D76FDD351E68D92AE4A3F3AEA7EC6A.epub 
	Paolo Bacigalupi
	Pump Six and Other Stories
DRM-free     Books/EEE65A68378C2510E18DF24CD767AC9A.epub 
	Fritz Leiber
	Swords Against Wizardry-Fafhrd and the Gray Mouser-Book 4
DRM-free     Books/F3BC14466A16C96CCD8FFE00DCCF8147.epub 
	Glen Cook
	An Empire Unacquainted With Defeat
DRM-free     Books/F534835595041374B814D151A633E69D.epub 
	Peter Watts
	Blindsight
DRM-free     Books/F8BF760284ADC800B290DCA6D8EA7EF2.epub 
	Ben Klemens
	21st Century C
DRM-free     Books/FE8C57863E40B98CB732FEE4BFDB60BB.epub 
	Glen Cook
	An Ill Fate Marshalling
8 books are DRM-infested
26 could be cured

Update (2013-11-06):

OS X 10.9 Mavericks and the new iBooks app changed the location of the iBooks directory, I changed my script accordingly (and made it adjust depending on which OS version you have). Also, the file names have changed and no longer embed author and title, so I am extracting them from the XML metadata files.

The Gresham’s law of Amazon Web Services

In the bad (good?) old days when currency’s worth was established by the amount of gold or silver in coinage, kings would cut corners by debasing currency with lead, which is almost as dense as gold or silver. In the New World, counterfeiters debased gold coins with platinum, which was first smelted by pre-columbian civilizations. Needless to say, the fakes are now worth more than the originals.

The public was not fooled, however, and found ways to test coins for purity, including folkloric ones like biting a coin to see if it is made of malleable gold, rather than harder metals. People would then hoard pure gold coins, and try to rid themselves of debased coins at the earliest opportunity. This led to Gresham’s Law: bad money drives out good money in circulation.

After a year of using Amazon Web Services’ EC2 service at scale for my company (we moved to our own servers at the end of 2011), I conjecture there is a Gresham’s Law of Amazon EC2 instances – bad instances drive out good ones. Let me elaborate:

Amazon EC2 is a good way to launch a service for a startup, without incurring heavy capital expenditures when getting started and prior to securing funding. Unfortunately, EC2 is not a quality service. Instances are unreliable (we used over 80 instances at Amazon, and there was at least one instance failure a week, and sometimes up to 4). Amazon instances have poor disk I/O performance that makes them particularly unsuitable to hosting non-trivial databases (EBS is even worse, and notoriously unreliable).

Performance is also inconsistent—I routinely observed “runt” m1.large instances that performed half as well as the others. We experienced all sorts of failure modes, including disk corruptions, disks that would block forever without timing out, sporadic losses of network connectivity, and many more. Even more puzzling, I would get 50% to 70% failure rate on new instances that would not come up cleanly after being launched.

Some of this is probably due to the fact we use an uncommon OS, OpenSolaris, that is barely supported on EC2, but I suspect a big part of this is that Amazon uses low-end commodity parts, and does not proactively retire failed or flaky hardware from service. Instances that have the bad luck of being assigned to flaky hardware are more likely to fail or perform poorly, and thus more likely to be be destroyed, released and a new one reassigned in the same slot. The inevitable consequence of this is that new instances have a higher likelihood of being runts or otherwise defective than long-running ones.

One work-around is to spin up a large number of instances, test them, and destroy the poor-performing ones. AWS runts are usually correlated with slower CPU clock speeds, as older machines would be running older versions of the Xen hypervisor Amazon uses under the hood, have less cache, slower drives and so on. Iterating through virtual machines as if you are picking melons at a supermarket is a slow and painful job, however, and even their newer machines have their share of runts. We were trying to keep only machines with 2.6 or 2.66GHz processors, but more than 70% of the instances we were getting assigned were 2.2GHz runts, and it would usually take creating 5 or 6 instances on average to get a non-runt.

In the end, we migrated to our own facility in colo, because Amazon’s costs, reliability and performance were just not acceptable, and we had long passed the threshold beyond which it is cheaper to own than rent (I estimate it at $5,000 to $10,000 per month Amazon spend, depending on your workload). It is not as if other cloud providers are any better—before Amazon we had started on Joyent, which supports OpenSolaris natively, and their MTBF was in the order of 2 weeks, apparently because they replaced their original Sun hardware with substandard Dell servers and had issues with power management C-states in the Dell server BIOS.

The dirty secret of cloud services is that there is no reliable source of information on actual performance and reliability of cloud services. This brings out another economic concept, George Akerlof’s famous paper on the market for lemons. In a market where information asymmetry exists, the market will eventually collapse in the absence of guarantees. Until Amazon and others offer SLAs with teeth, you should remain skeptical about their ability to deliver on their promises.

Withings smart baby monitor review

One of the joys challenges of being a first-time parent is being exposed to a bewildering array of gadgets and equipment required to care for the baby, from baby car seats, strollers and diaper pails to 2-axis rocking robots (thanks Rohit!). There is an entire cottage industry of books like Baby Bargains that help you navigate through the confusing and sometimes questionable or outright unnecessary choices.

I have a Withings body weight scale that I really like and I was excited to learn they were going to release a networked video baby monitor. It took a while to get to market in the US, however, so in the interim I purchased a Philips Advent DECT digital baby monitor, which ended up unusable in practice, because its microphone sensitivity is so poor that you can barely hear anything. When the Withings baby monitor finally became available in the US, I immediately ordered it.

Withings is clearly taking design cues from Apple, from the lavishly designed packaging to the glossy white plastic RoundedRect aesthetic and the use of a magnetic clip to attach the baby monitor to the crib. The clip is serviceable, but the magnets are not quite strong enough to hold the unit firmly onto the crib. I would not trust it to keep the monitor from toppling when the baby grows and kicks at the crib. Fortunately they also include a flip-out tab on the base of the unit that can be inserted into a slit on the clip to prevent sliding, although it is not obvious and it took me a while before I discovered this key feature.

The wall wart is a generic black model with swappable AC prongs for international markets, and detracts from the overall package, but since the monitor has a micro-USB input, you can always use another standard AC to USB type A adapter like the iPhone’s, with a USB type A to micro-USB cable. A rechargeable battery is included, with 2 hours’ claimed life, I did not verify that spec.

The initial out of the box experience is good: you connect to the device from your iPhone or iPad using Bluetooth (no messing around with a USB cable as with the Withings scale), enter the WiFi settings in the Withbaby app, and then use WiFi to access the device afterwards. It is as streamlined an experience as you can expect without a keyboard on the unit. There is also an Ethernet jack (it is unclear whether it supports power over Ethernet), but my house was built in 1928 and is not wired upstairs where the baby lives.

Once you enter your credentials into the app, it connects to the monitor and shows you the video and sound. If you put it in the background, you have the option of monitoring audio. Withings will also send you alerts via push notifications if the temperature or humidity is excessive, or if it detects noise or motion. The default settings are way too twitchy, however, and you will find yourself disabling audio notifications as the deluge of alerts is just too much.

The device includes a night light with selectable color, a lullaby player, and the ability to speak to your baby, all controlled through the app. At the front you also have touch controls to turn some of these features on. This is actually a bad idea, as on two occasions I started the lullaby by accident as I was fumbling with it in a dark room, and woke up my baby as a result. Another design flaw is the pulsing blue night light when the unit is rebooting, the Airport Express like amber/green status LED in the back is quite sufficient. Frankly the only one of these features that is useful is the speaker, and the ability to stream from your music collection, such as Dr. Harvey Karp’s white noise selections would be preferable to the canned lullabies.

The video camera is advertised as having a 3 megapixel sensor. It has a wide-angle lens and you can “pan” using the usual iPhone or iPad gestures. The lens is a fixed-focus plastic one, and optical clarity is so-so at best, optimal focus seems to be at 50cm or so. One great feature is the monitor has a normal and night vision mode, similar to the one on some Sony HAD camcorders, with an IR illuminator that provides light for the night vision mode. This means you can watch your baby toss and turn in an otherwise pitch-black room.

You can use the baby monitor from outside your network, and it works fine, even over a 3G connection. Withings allows you up to 15 minutes per day, anything beyond that requires paying them $6 for each 100 minutes. Coming on top of an already expensive device, this seems like a naked money grab from anxious parents. (Updated 2012-09-29: remote monitoring is now free and unlimited).

When the unit works, it is absolutely great: good sound sensitivity and the video feature mostly works as advertised. Unfortunately it frequently does not function, and I find myself performing a hard reboot by removing the battery far more often than I would like. Among the pathologies:

  • Once it falsely reported the unit was closed and thus video inaccessible
  • Once the camera was in a frozen state, it took a power cycling to get the video moving again.
  • Yesterday I could not connect at all, no matter how many times I rebooted my Airport Extreme, the monitor and my wife’s or my iPads. Some detective work using a packet sniffer showed the app was trying to connect to babyws.withings.net using HTTP, which is aliased to s11.withings.net, and that server was down. Some of the documentation suggests you can use the Bluetooth connection to access the monitor, but I was not able to figure out how to do this.

This brings me to a crucial point. The baby monitor is a safety device, and it is utterly unacceptable for its functioning to be dependent on a cloud service, which can and will be a single point of failure. It should use Bonjour or similar discovery methods to work on the LAN, and rely on Withings’ servers only when accessing it from outside the home LAN’s perimeter. I wonder if Withings’ eagerness to nickel-and-dime users by charging for outside monitoring led to this critical design flaw.

The bottom line is the Withings smart baby monitor is a very frustrating device, with its obvious potential marred by failures of execution. If it worked consistently, it would be a top-notch product worthy of its Apple inspiration and lofty price tag, but the general lack of reliability means I cannot recommend it until the bugs are ironed out. Consider it an alpha release at best.

Update (2012-09-17):

Here’s how to make the Withings not-so-smart baby monitor more usable:

  • Remove the battery from the unit and hook up the micro-USB power adapter to a Belkin WeMo remote-controlled power switch. This allows you to power-cycle the baby monitor remotely from the same iPhone or iPad you are using the monitor software on.
  • Hide the blue led with gaffer’s tape. This prevents the blue light on reboot from waking the baby. Unlike duct tape, gaffer’s tape can be removed without leaving glue residue, although the aesthetics of dark gray gaffer’s tape on the gleaming white unit are questionable at best.
  • I haven’t tried covering up the touch controls with gaffer’s tape, which would eliminate the risk of triggering a jingle and waking the baby. The WeMo eliminates the need to enter the room and tinker with the baby monitor.

It’s quite sad to have to pay an extra $50 to work around buggy hardware and software, but it makes a big difference.

Update (2013-05-20):

The micro-USB connector failed and the baby monitor is now essentially a doorstop. Not surprising given how flimsy micro-USB is, compared to mini-USB, for insignificant space savings. Micro-USB was a Nokia design rammed through the USB-IF. In theory it has better insert-remove cycle life than mini-USB, but in practice I’ve never had mini-USB fail, whereas it is a frequent occurrence with micro-USB.

Update (2015-08-17):

USB-C is an improvement over micro-USB, hopefully some future version of the baby monitor will use it. Still nowhere near as robust as Lightning or tip-ring-sleeve, though.

The HP-15C was reissued at long last!

It is strange no one seems to have picked up the news yet, but HP has reissued the legendary HP-15C in a special “30th anniversary limited edition”, and it became available for purchase last week.

HP-15C Limited Edition

The new HP-15C is not strictly speaking a reissue but a replica, as it does not use the original’s Saturn processor, but instead an emulation thereof running on an ARM CPU. Even emulated, it should be much faster than the original 640 kilohertz processor. I ordered two, and received them today.

As expected, the quality is in line with the current HP-12C, i.e. not as good as the 1980s models in terms of key feel, but still leagues ahead of any competing product. The originals used a special 47-point bonding process to ensure the utmost in rigidity and reliability, I doubt the current model had as much attention paid to detail. It is made in China, obviously, the Corvallis facility is long gone. The slipcover fits very poorly (too tight, and the seams are not trimmed properly) and feels thinner and outright cheap compared to the original. The labels on the keys are accurately positioned, at least, unlike the train wreck that was the HP-12C Platinum. The cheat sheet in the back is a garish black on silver as on the 35S, instead of the original’s silver on black. It also uses two 3V CR2032 batteries instead of the 3 button cells in  the original.

Speed-wise, the Limited Edition integrates the normal distribution nearly instantly, when that test that took 34 seconds on the original.

In short: not as good as the original, but still an excellent calculator for those who prize ergonomics.

A waiter for a server

I had to monitor a long-running process on a Solaris server tonight, but didn’t want to stay glued at a computer monitor. A neat trick:

ssh myserver.example.com "pwait 17601"; say "batch done"

You would replace 17601 with the process ID of the job you are waiting for, of course. That way, my Mac connects to the server, waits for the job to complete, then gives me an spoken alert when it is done. I can watch a movie, do chores or whatever during that time. I am sure there are equivalent commands to pwait for Linux.

Will Adobe ever learn?

In a triumph of hope over experience, I recently “upgraded” from Adobe CS3 Design Standard to CS5 Design Standard. I hardly ever use Photoshop any more since I started using Aperture and Lightroom (originally a Macromedia product, no matter what the lame “Adobe Photoshop Lightroom” face-saving branding may try to claim), the main driver for the purchase was actually InDesign CS5 and its ePub functions.

Of course, this is Adobe. Previous versions gratuitously included crud like a full Opera install (an older version, insecure, naturally) just to display a splash screen, or a full MySQL install to power Acrobat search. I never install Acrobat, of course, since that bloated and bug-ridden piece of garbage managed to steal the crown for most insecure software from Internet Explorer, no small feat.

Adobe does not want to confuse users with streamlined and efficient software, so they decided to include the mostly useless Growl on-screen notification program to nag you into registering. Increasing bloat and attack surface for malware is not a good idea, nor is interrupting creative people’s flow with interruptions. Of course, helping clients Get Things Done is a low priority at Adobe, as evidenced by their product choices.

You have to pity the Growl developers, whose reputation will suffer from guilt by association. I dislike interruptions and do not find it useful, but many people do and rave about it. They installed it by choice, not as a sneaky drive-by install for slimy marketing purposes.

Some more annoyances in CS5:

  • The pricing for the suite is more than the sum of its parts: $200 each for Photoshop, InDesign or Illustrator, $700 for Design Standard. I suppose they must think Acrobat and their online tie-ins have a value of $100 (hint: they forgot the negative sign).
  • Of course, they won’t let you upgrade individual component applications.
  • On the plus side, they now have the decency to include Acrobat on a separate CD, so you can discard it immediately and not risk installing it as a side-effect of installing the apps that are actually useful.
  • The icons were designed by the world’s laziest and most creatively bankrupt designer, just as with CS3 and CS4
  • Performance on a high-end 8-core or 12-core Mac is actually slower than on a lower-end configuration, thanks to legacy cruft and incompetence.
  • It is slower to load on my wife’s MacBook Pro. Each successive version of OS X is faster on the same hardware, Microsoft and Adobe deliver software that gets progressively slower.

In other words, unlike Lightroom, CS5 is designed to be endured, not to delight.

Incensed at Mozilla

One of the greatest features in the Webkit-based browsers (Apple’s Safari and Google Chrome) is WebSQLdatabase, the ability for a web site to store information in a SQLite database on your browser accessible via JavaScript. This allows web developers to build database-enabled applications that run entirely in the browser, without requiring a server. This is very useful for mobile devices, which in the US enjoy flaky network connectivity at best. One very handsome example is the iPad-optimized Every Time Zone webapp.

SQLite is probably the most important open-source project you have never heard of. It is a simple, streamlined and efficient embedded database. Firefox stores its bookmarks in it. Google distributes its database of phishing sites in that format. Sun’s industrial-strength Solaris operating system stores the list of services it runs on boot in it—if it were to fail, a server would be crippled so that is a pretty strong vote of confidence. Adobe Lightroom and Apple’s Aperture use it to store their database, as do most Mac applications that use the CoreData framework, and many iPhone apps. In other words, it is robust and proven mission-critical software that is widely yet invisibly deployed.

WebSQLdatabase basically makes the power of SQLite available to web developers trying to build apps that work offline, specially on mobile devices. No good deed goes unpunished, and the Mozilla foundation teamed up with unlikely bedfellow Microsoft to torpedo formal adoption of WebSQLdatabase as a web standard, on spurious grounds, and pushed an alternate standard called IndexedDB instead. To quote the Chromium team:

Q: Why this over WebSQLDatabase?

A: Microsoft and Mozilla have made it very clear they will not implement SQL in the browser.  If you want to argue this is silly, talk to them, not me.

IndexedDB is several steps backwards. Instead of using powerful, expressive and mature SQL technology, it uses a verbose JavaScript B-tree API that is a throwback to the 1960s bad old days of hierarchical databases and ISAM, requires a lot more work from the developer, for no good reason. To add injury to insult, Firefox 4’s implementation of IndexedDB is actually built on top of SQLite. The end result will be that web developers will need to build a SQL emulation library on top of IndexedDB to restore the SQLite functionality deliberately crippled by IndexedDB. If there is one constant in software engineering, it is that multiple layers add brittleness and impair performance.

Of course, both Mozilla and Microsoft are irrelevant on mobiles, where WebKit has essentially won the day, so why should this matter? Microsoft has always been a hindrance to the development of the web, since they have to protect the Windows API from competition by increasingly capable webapps, but I cannot understand Mozilla’s attitude, except possibly knee-jerk not-invented-here syndrome and petulance at being upstaged by WebKit. WebSQLdatabase is not perfect—to reach its full potential, it needs and automatic replication and sync facility between the local database and the website’s own database, but it is light years ahead of IndexedDB in terms of power and productivity.

I am so irritated by Mozilla’s attitude that after 10 years of using Mozilla-based browsers, I switched today from Firefox to Chrome as my primary browser. Migrating was surprisingly easy. Key functionality like bookmark keywords, AdBlock, FlashBlock, a developer console, and the ability to whitelist domains for cookies, all have equivalents on Chrome. The main regressions are bookmark tags, and Chrome’s sync options are not yet equivalent to Weave‘s. At some point I will need to roll my own password syncing facility (Chrome stores its passwords in the OS X keychain, which is also used by Safari and Camino).

RapidSSL 1 – GoDaddy 0

My new company’s website uses SSL. I ordered an “extended validation” certificate from GoDaddy, instead of my usual CA, RapidSSL/GeoTrust, because GoDaddy’s EV certificates were cheap. EV certificates are security theater more than anything else, I probably should not have bothered.

Immediately after switching from my earlier “snake oil” self-signed test certificate to the production certificate, I saw SSL errors on Google Chrome for Mac and Safari for Mac, i.e. the two browsers that use OS X’s built-in crypto and certificate store. I suppose I should have tested the certificate on another server before going live, but I trusted GoDaddy (they are my DNS registrars, and competent, if garish).

Big mistake.

I called their tech support hotline, which is incredibly grating because of the verbose phone tree that keeps trying to push add-ons (I guess it is consistent with the monstrosity that is their home page).

After a while, I got a first-level tech. He asked whether I saw the certificate error on Google Chrome for Windows. At that point, I was irate enough to use a four-letter word. Our customers are Android mobile app developers. A significant chunk of them use Macs, and almost none (less than 5%) use IE, so know-nothing “All the world is IE” demographics are not exactly applicable.

After about half an hour of getting the run-around and escalating to level 2, with my business partner Michael getting progressively more anxious in the background, the level 1 CSR tells me the level 2 one can’t reproduce the problem (I reproduced it on three different Macs in two different locations). I gave them an ultimatum: fix it within 10 minutes or I would switch. At this point, the L1 CSR told me he had exhausted all his options, but I could call their “RA” department, and offered to switch me. Inevitably, the call transfer failed.

I dialed their SSL number, and in parallel started the certificate application process on RapidSSL. They offered a free competitive upgrade, I tried it, and within 3 minutes I had my fresh new, and functional certificate, valid for 3 years, all for free and in less time than it takes to listen to GoDaddy’s obnoxious phone tree (all about “we pride ourselves in customer service” and other Orwellian corporate babble).

I then called GoDaddy’s billing department to get a refund. Surprisingly, the process was very fast and smooth. I guess it is well-trod.

The moral of the story: GoDaddy—bad. RapidSSL—good.

Update (2012-08-26)

I switched my DNS business from GoDaddy to Gandi.net in December 2011 after Bob Parsons’ despicable elephant-hunting stunt.

One month with the iPad

Since I got my iPad six weeks ago, I have only used my MacBook Air once.

I am not going to repeat the extensive reviews posted elsewhere, but after over a month of extensive use, give some perspective for those who don’t get the point of the iPad, or other similar devices.

First of all, commentators have focused on entirely the wrong thing: feeds and speeds, missing features like multitasking or Flash, Apple’s iron fist over app developers. The iPad begins and ends with the user experience, and that means multi-touch and the incredibly long battery life. That’s why comparisons to stylus-driven devices like the unsuccessful Microsoft Tablet PC miss the point. The amazing battery life, specially on standby (I have never managed to go under 60%, even after three days without charging), means you can use it as a real mobile device and not subconsciously watch the battery meter.

Is it a perfect device? Of course not. Mobile Safari has a hard time with complex and heavy pages like those from my Temboz RSS/Atom feed reader, the screen is too prone to reflections and fingerprints, and Apple’s use of high-quality materials like aluminium and glass instead of plastic and acrylic makes it heavier to hold than necessary.

As to whether it is a replacement for a laptop, the answer is yes and no. The iPad is the first in an entirely new class of devices, and I think it has the potential to replace desktop and laptop computers as the dominant form of consumer computing. The touch user interface makes for a very engaging user experience, far more than using a mouse and keyboard ever did. To be sure, the input limitations do not make it a very efficient content creation device, but that’s where opinions diverge.

I use desktop computers for real work (an eight-core Mac Pro with 12G of RAM and a 30″ display at home, a quad-core iMac with a 27″ display at work). A laptop just feels too constricting for extended use. I have the luxury of using proper desktops because I do not travel much for work, and the extent of my mobile use is reading books or browsing the web while commuting by bus. The improvements that most benefit me are in synchronizing my iPad with multiple computers, and offline capability (I got the WiFi model since there is no way I will pay AT&T for their garbage excuse of a network).

Road warriors need a more featured device, even if cramped, and will not be so impressed. I think genuine mobile users are a minority, however. Surveys in the past showed that most laptops are tethered, i.e. users would unplug them from home, take them to work and plug them there, and back. That is why Windows laptop makers introduced monstrosities like Pentium 4 powered laptops with battery lives that barely exceeded the hour. Laptop sales exceeded those of desktops because many people wanted the option of mobility, even if they seldom, if ever, availed themselves of it, and a less obtrusive presence in their homes than the typical beige box with its rat’s warren of cables. Those people would be better served by a well-designed desktop like the iMac and an iPad for the occasional mobile use.

Clueless SaaS providers can leave you with egg on your face

While cleaning out my spam folders, I noticed a disturbing trend: a number of the spam were sent to vendor-specific email addresses I had set up to communicate with Parallels, Joyent and Shoeboxed. As a security measure, I do not give my personal email address to vendors, only aliases. The email address I used in the past for Dell was dell@majid.fm, for instance (I now use a different domain). A few years back, I started receiving pornographic spam at that address, which led me to think either Dell had secretly adopted a radically new diversification plan, or that their customer database had been compromised. Needless to say, this did not reflect well on Dell. I canceled that alias and stopped dealing with Dell.

I contacted the support for the three vendors. Joyent got back to me, and said:

We have traced this back to a third-party provider that was used to distribute service notifications. We have been in contact with this service provider, and they have determined that subscriber email addresses of their clients were compromised. They have launched their own investigation, which is ongoing, and have also reached out to their local FBI office.

After some digging, I found some interesting posts. Some email marketing company called iContact, that I had never heard about before, was the source of the compromise. They claim to be SAS-70 compliant, but of course like most bureaucratic certifications, SAS-70 is mostly security theater that makes sysadmins’ life miserable for no meaningful security benefit (SAS-70 auditors, on the other hand, profit handsomely).

Just another example of how outsourcing critical functions to outside vendors can backfire spectacularly and take down your own reputation in the process.

Just enough Weave

Note: I am keeping this code around for historical purposes, but it has not worked since Weave 1.0 RC2. I created this because Mozilla’s public sync servers were initially quite unreliable, but they have remedied the situation and performance problems are a thing of the past. I also learned the inner workings of Weave/Firefox Sync in the process, and am satisfied as to the security of the system. Since I no longer use Firefox myself, I do not expect to ever revive this project. Feel free to take it over, otherwise you are best served by using Mozilla’s cloud.

Like most of my readers, I use multiple computers: my Mac Pro at home, my MacBook Air when on the road, 3 desktop PCs at work, a number of virtual machines, and so on. I have Firefox installed on all of them. The Mozilla Weave extension allows me to sync bookmarks, passwords et al between them. Weave encrypts this data before uploading it to the server, but I do not like to rely on third-party web services for mission-critical functions (my Mozilla server was down last Monday, for instance, due to the surge of traffic from people returning to work and performing a full sync against 0.5). Through Weave 0.5, I ran my own instance of the Mozilla public Weave server version 0.3. Unfortunately, Weave 0.6 requires server version 0.5 and I had to upgrade.

The open-source Weave server is implemented in PHP. It doesn’t require Apache compiled with mod_dav as early versions did (I prefer to run nginx), but it is still a fairly gnarly piece of code that is anything but plug-and-play. Somehow I had managed to get version 0.3 running on my home server, but no amount of blundering around got me to a usable state with 0.5. I ended up deciding to implement a minimalist Weave server in Python, as it seemed less painful than continuing to struggle with the Mozilla spaghetti code, which confusingly features multiple pieces of code that appear to do exactly the same thing in three different places. Famous last words…

Three days of hacking later, I managed to get it working. 200 or so lines of Python code replaced approximately 12,000 lines of PHP. Of course, I am not trying to reproduce an entire public cloud infrastructure like Mozilla’s, just enough for my own needs, using the “simplest thing that works” principle. Interestingly, the Mozilla code includes a vestigial Python reference implementation of a Weave server for testing purposes. It does not seem to have been working for a while, though. I used it as a starting point but ended up rewriting almost everything. Here are the simplifying hypotheses:

  • My weave server is meant for a single user (my wife prefers Safari)
  • It does not implement authentication, logging or SSL encryption — it is meant to be used behind a nginx (or Apache) reverse proxy that will perform these functions.
  • It has no configuration file. There are just three variables to set at the top of the source file.
  • It does not implement the full server protocol, just the parts that are actually used by the extension today.
  • More controversially, it does not even implement persistence, keeping all data in RAM instead. Python running on Solaris is very reliable, and the expected uptime of the server is likely months on end. If the server fails, the Firefoxes will just have to perform a full sync and reconciliation. Fortunately, that has been much improved in Weave 0.6, so the cost is minimal. This could even be construed as a security feature, since there is no data on disk to be misplaced. It would take catastrophically losing all my browsers simultaneously to risk data loss. Short of California falling into the ocean, that’s not going to happen, and if it does, I probably have more pressing concerns…

The code could be extended fairly easily to lift these hypotheses, e.g. adding persistence or multiple user support using SQLite, PostgreSQL or MySQL.

Here is the server itself, weave_server.py:

#!/usr/local/bin/python
"""
  Based on tools/scripts/weave_server.py from
  http://hg.mozilla.org/labs/weave/

  do the Simplest Thing That Can Work: just enough to get by with Weave 0.6
  - SSL, authentication and loggin are done by nginx or other reverse proxy
  - no persistence, in case of process failure do a full resync
  - only one user. If you need more, create multiple instances on different
    ports and use rewrite rules to route traffic to the right one
"""

import sys, time, logging, socket, urlparse, httplib, pprint
try:
  import simplejson as json
except ImportError:
  import json
import wsgiref.simple_server

URL_BASE = 'https://your.server.name/'
#BIND_IP = ''
BIND_IP = '127.0.0.1'
DEFAULT_PORT = 8000

class HttpResponse:
  def __init__(self, code, content='', content_type='text/plain'):
    self.status = '%s %s' % (code, httplib.responses.get(code, ''))
    self.headers = [('Content-type', content_type),
                    ('X-Weave-Timestamp', str(timestamp()))]
    self.content = content or self.status

def JsonResponse(value):
  return HttpResponse(httplib.OK, value, content_type='application/json')

class HttpRequest:
  def __init__(self, environ):
    self.environ = environ
    content_length = environ.get('CONTENT_LENGTH')
    if content_length:
      stream = environ['wsgi.input']
      self.contents = stream.read(int(content_length))
    else:
      self.contents = ''

def timestamp():
  # Weave rounds to 2 digits and so must we, otherwise rounding errors will
  # influence the "newer" and "older" modifiers
  return round(time.time(), 2)

class WeaveApp():
  """WSGI app for the Weave server"""
  def __init__(self):
    self.collections = {}

  def url_base(self):
    """XXX should derive this automagically from self.request.environ"""
    return URL_BASE

  def ts_col(self, col):
    self.collections.setdefault('timestamps', {})[col] = str(timestamp())

  def parse_url(self, path):
    if not path.startswith('/0.5/') and not path.startswith('/1.0/'):
      return
    command, args = path.split('/', 4)[3:]
    return command, args

  def opts_test(self, opts):
    if 'older' in opts:
      return float(opts['older'][0]).__ge__
    elif 'newer' in opts:
      return float(opts['newer'][0]).__le__
    else:
      return lambda x: True

  # HTTP method handlers

  def _handle_PUT(self, path, environ):
    command, args = self.parse_url(path)
    col, key = args.split('/', 1)
    assert command == 'storage'
    val = self.request.contents
    if val[0] == '{':
      val = json.loads(val)
      val['modified'] = timestamp()
      val = json.dumps(val, sort_keys=True)
    self.collections.setdefault(col, {})[key] = val
    self.ts_col(col)
    return HttpResponse(httplib.OK)

  def _handle_POST(self, path, environ):
    try:
      status = httplib.NOT_FOUND
      if path.startswith('/0.5/') or path.startswith('/1.0/'):
        command, args = self.parse_url(path)
        col = args.split('/')[0]
        vals = json.loads(self.request.contents)
        for val in vals:
          val['modified'] = timestamp()
          self.collections.setdefault(col, {})[val['id']] = json.dumps(val)
        self.ts_col(col)
        status = httplib.OK
    finally:
      return HttpResponse(status)

  def _handle_DELETE(self, path, environ):
    assert path.startswith('/0.5/') or path.startswith('/1.0/')
    response = HttpResponse(httplib.OK)
    if path.endswith('/storage/0'):
      self.collections.clear()
    elif path.startswith('/0.5/') or path.startswith('/1.0/'):
      command, args = self.parse_url(path)
      col, key = args.split('/', 1)
      if not key:
        opts = urlparse.parse_qs(environ['QUERY_STRING'])
        test = self.opts_test(opts)
        col = self.collections.setdefault(col, {})
        for key in col.keys():
          if test(json.loads(col[key]).get('modified', 0)):
            logging.info('DELETE %s key %s' % (path, key))
            del col[key]
      else:
        try:
          del self.collections[col][key]
        except KeyError:
          return HttpResponse(httplib.NOT_FOUND)
    return response

  def _handle_GET(self, path, environ):
    if path.startswith('/0.5/') or path.startswith('/1.0/'):
      command, args = self.parse_url(path)
      return self.handle_storage(command, args, path, environ)
    elif path.startswith('/1/'):
      return HttpResponse(httplib.OK, self.url_base())
    elif path.startswith('/state'):
      return HttpResponse(httplib.OK, pprint.pformat(self.collections))
    else:
      return HttpResponse(httplib.NOT_FOUND)

  def handle_storage(self, command, args, path, environ):
    if command == 'info':
      if args == 'collections':
        return JsonResponse(json.dumps(self.collections.get('timestamps', {})))
    if command == 'storage':
      if '/' in args:
        col, key = args.split('/')
      else:
        col, key = args, None
      try:
        if not key: # list output requested
          opts = urlparse.parse_qs(environ['QUERY_STRING'])
          test = self.opts_test(opts)
          result = []
          for val in self.collections.setdefault(col, {}).itervalues():
            val = json.loads(val)
            if test(val.get('modified', 0)):
              result.append(val)
          result = sorted(result,
                          key=lambda val: (val.get('sortindex'),
                                           val.get('modified')),
                          reverse=True)
          if 'limit' in opts:
            result = result[:int(opts['limit'][0])]
          logging.info('result set len = %d' % len(result))
          if 'application/newlines' in environ.get('HTTP_ACCEPT', ''):
            value = '\n'.join(json.dumps(val) for val in result)
            return HttpResponse(httplib.OK, value,
                                content_type='application/text')
          else:
            return JsonResponse(json.dumps(result))
        else:
          return JsonResponse(self.collections.setdefault(col, {})[key])
      except KeyError:
        if not key: raise
        return HttpResponse(httplib.NOT_FOUND, '"record not found"',
                            content_type='application/json')

  def __process_handler(self, handler):
    path = self.request.environ['PATH_INFO']
    response = handler(path, self.request.environ)
    return response

  def __call__(self, environ, start_response):
    """Main WSGI application method"""

    self.request = HttpRequest(environ)
    method = '_handle_%s' % environ['REQUEST_METHOD']

    # See if we have a method called 'handle_METHOD', where
    # METHOD is the name of the HTTP method to call.  If we do,
    # then call it.
    if hasattr(self, method):
      handler = getattr(self, method)
      response = self.__process_handler(handler)
    else:
      response = HttpResponse(httplib.METHOD_NOT_ALLOWED,
                              'Method %s is not yet implemented.' % method)

    start_response(response.status, response.headers)
    return [response.content]

class NoLogging(wsgiref.simple_server.WSGIRequestHandler):
  def log_request(self, *args):
    pass

if __name__ == '__main__':
  socket.setdefaulttimeout(300)
  if '-v' in sys.argv:
    logging.basicConfig(level=logging.DEBUG)
    handler_class = wsgiref.simple_server.WSGIRequestHandler
  else:
    logging.basicConfig(level=logging.ERROR)
    handler_class = NoLogging
  logging.info('Serving on port %d.' % DEFAULT_PORT)
  app = WeaveApp()
  httpd = wsgiref.simple_server.make_server(BIND_IP, DEFAULT_PORT, app,
                                            handler_class=handler_class)
  httpd.serve_forever()

Here is the relevant fragment from my nginx configuration file:

# Mozilla Weave
location /0.5 {
  auth_basic            "Weave";
  auth_basic_user_file  /home/majid/web/conf/htpasswd.weave;
  proxy_pass            http://localhost:8000;
  proxy_set_header      Host $http_host;
}
location /1.0 {
  auth_basic            "Weave";
  auth_basic_user_file  /home/majid/web/conf/htpasswd.weave;
  proxy_pass            http://localhost:8000;
  proxy_set_header      Host $http_host;
}
location /1/ {
  auth_basic            "Weave";
  auth_basic_user_file  /home/majid/web/conf/htpasswd.weave;
  proxy_pass            http://localhost:8000;
  proxy_set_header      Host $http_host;
}

This code is hereby released into the public domain. You are welcome to use it as you wish. Just keep in mind that since it is reverse-engineered, it may well break with future releases of the Weave extension, or if Mozilla changes the server protocol.

Update (2009-10-03):

I implemented some minor changes for compatibility with Weave 0.7. The diff with the previous version is as follows:

--- weave_server.py~	Thu Sep  3 17:46:44 2009
+++ weave_server.py	Sat Oct  3 02:59:19 2009
@@ -65,8 +65,7 @@
     command, args = path.split('/', 4)[3:]
     return command, args

-  def opts_test(self, environ):
-    opts = urlparse.parse_qs(environ['QUERY_STRING'])
+  def opts_test(self, opts):
     if 'older' in opts:
       return float(opts['older'][0]).__ge__
     elif 'newer' in opts:
@@ -92,7 +91,7 @@
   def _handle_POST(self, path, environ):
     try:
       status = httplib.NOT_FOUND
-      if path.startswith('/0.5/') and path.endswith('/'):
+      if path.startswith('/0.5/'):
         command, args = self.parse_url(path)
         col = args.split('/')[0]
         vals = json.loads(self.request.contents)
@@ -113,7 +112,8 @@
       command, args = self.parse_url(path)
       col, key = args.split('/', 1)
       if not key:
-        test = self.opts_test(environ)
+        opts = urlparse.parse_qs(environ['QUERY_STRING'])
+        test = self.opts_test(opts)
         col = self.collections.setdefault(col, {})
         for key in col.keys():
           if test(json.loads(col[key]).get('modified', 0)):
@@ -142,10 +142,14 @@
       if args == 'collections':
         return JsonResponse(json.dumps(self.collections.get('timestamps', {})))
     if command == 'storage':
-      col, key = args.split('/')
+      if '/' in args:
+        col, key = args.split('/')
+      else:
+        col, key = args, None
       try:
         if not key: # list output requested
-          test = self.opts_test(environ)
+          opts = urlparse.parse_qs(environ['QUERY_STRING'])
+          test = self.opts_test(opts)
           result = []
           for val in self.collections.setdefault(col, {}).itervalues():
             val = json.loads(val)
@@ -155,6 +159,8 @@
                           key=lambda val: (val.get('sortindex'),
                                            val.get('modified')),
                           reverse=True)
+          if 'limit' in opts:
+            result = result[:int(opts['limit'][0])]
           logging.info('result set len = %d' % len(result))
           if 'application/newlines' in environ.get('HTTP_ACCEPT', ''):
             value = '\n'.join(json.dumps(val) for val in result)

Update (2009-11-17):

Weave 1.0b1 uses 1.0 as the protocol version string instead of 0.5 but is otherwise unchanged. I updated the script and nginx configuration accordingly.

Diminishing returns

I have an eight-core Nehalem Mac Pro. Most of these cores sit idle most of the time due to poorly written software that is not optimized for the post-Moore multicore world.

I am beginning to wonder if Intel’s transistor budget wouldn’t be better allocated to more SRAM cache instead of more cores. One SRAM bit uses up 4 transistors, the Xeon 5500 have 751 million transistors, of which 8Mx8x4 or 256 million are for the 8MB L3 cache. If the chip were brought down from quad-core to dual-core, that would allow doubling the cache. Many programs could run entirely from cache, including interpreters.