Fazal Majid's low-intensity blog

Sporadic pontification

Automating Epson SSL/TLS certificate renewal

Network-capable Epson printers like my new ET-16600 have a web-based user interface that supports HTTPS. You can even upload publicly recognized certificates from Let’s Encrypt et al, unfortunately the only options they offer is a Windows management app (blech) or a manual form.

When you have to upload this every month (that’s when I automatically renew my Let’s Encrypt certificates), this gets old really fast, and strange errors happen if you forget to do so and end up with an expired certificate.

I wrote a quick Python script to automate this (and yes, I am aware of the XKCDs on the subject of runaway automation):

#!/usr/bin/env python3
import requests, html5lib

# update these fields for your environment
URL = 'https://myepson.example.com/'
USERNAME = 'majid'
PASSWORD = 'your-admin-UI-password-here'
KEYFILE = '/home/majid/web/acme-tiny/epson.key'
CERTFILE = '/home/majid/web/acme-tiny/epson.crt'
CAFILE = '/home/majid/web/acme-tiny/lets-encrypt-r3-cross-signed.pem'

# step 1, authenticate
jar = requests.cookies.RequestsCookieJar()
set_url = URL + 'PRESENTATION/ADVANCED/PASSWORD/SET'
r = requests.post(set_url, cookies=jar,
                  data={
                    'INPUTT_USERNAME': USERNAME,
                    'access': 'https',
                    'INPUTT_PASSWORD': PASSWORD,
                    'INPUTT_ACCSESSMETHOD': 0,
                    'INPUTT_DUMMY': ''
                  })
assert r.status_code == 200
jar = r.cookies

# step 2, get the cert update form iframe and its token
form_url = URL + 'PRESENTATION/ADVANCED/NWS_CERT_SSLTLS/CA_IMPORT'
r = requests.get(form_url, cookies=jar)
tree = html5lib.parse(r.text, namespaceHTMLElements=False)
data = dict([(f.attrib['name'], f.attrib['value']) for f in
             tree.findall('.//input')])
assert 'INPUTT_SETUPTOKEN' in data

# step 3, upload key and certs
data['format'] = 'pem_der'
del data['cert0']
del data['cert1']
del data['cert2']
del data['key']

upload_url = URL + 'PRESENTATIONEX/CERT/IMPORT_CHAIN'
r = requests.post(upload_url, cookies=jar,
                  files = {
                    'key': open(KEYFILE, 'rb'),
                    'cert0': open(CERTFILE, 'rb'),
                    'cert1': open(CAFILE, 'rb')
                  },
                  data=data)

assert 'Shutting down' in r.text
print('Epson certificate successfully uploaded to printer.')

Update (2020-12-29):

If you are having problems with the Scan to Email feature, with the singularly unhelpful message “Check your network or WiFi connection”, it may be the Epson does not recognize the new Let’s Encrypt R3 CA certificate. You can address this by importing it in the Web UI, under the “Network Security” tab, then “CA Certificate” menu item on the left. The errors I was seeing in my postfix logs were:

Dec 29 13:30:20 zulfiqar mail.info postfix/smtpd[13361]: connect from epson.majid.org[10.0.4.33]
Dec 29 13:30:20 zulfiqar mail.info postfix/smtpd[13361]: SSL_accept error from epson.majid.org[10.0.4.33]: -1
Dec 29 13:30:20 zulfiqar mail.warn postfix/smtpd[13361]: warning: TLS library problem: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca:ssl/record/rec_layer_s3.c:1543:SSL alert number 48:
Dec 29 13:30:20 zulfiqar mail.info postfix/smtpd[13361]: lost connection after STARTTLS from epson.majid.org[10.0.4.33]
Dec 29 13:30:20 zulfiqar mail.info postfix/smtpd[13361]: disconnect from epson.majid.org[10.0.4.33] ehlo=1 starttls=0/1 commands=1/2

Apple iCalendar's buggy SNI

TL:DR If you use Apple’s calendar client software, do not run the server on an IP and port shared with any other SSL/TLS services.

I run my own CalDAV calendar server for my family and for myself. For a very long time I used DAViCal, but it’s always been a slight annoyance to set up on Apple devices because they don’t like DAViCal’s https://example.com/davical/caldav.php/majid URLs. What’s more, recent versions of iCalendar would pop up password prompts at random, and after re-entering the password a couple of times (once is not enough), would finally go on and work. The various devices would also all too often get out of sync, sometimes with the inscrutable error:

Server responded with “500” to operation CalDAVAccountRefreshQueueableOperation

requiring deleting the calendar account and recreating it by hand.

I tried replacing DAViCal with Radicale today, with the same flaky user experience, and I finally figured out why: Apple uses at least a couple of daemons to manage calendar and sync, including dataaccessd, accountsd and remindd (also CalendarAgent depending on your OS version). It seems some or all of them do not implement Server Name Indication (SNI) consistently. SNI is the mechanism by which a TLS client indicates what server it is trying to connect to during the TLS handshake, so multiple servers can share the same IP address and port, and is an absolutely vital part of the modern web. For example many servers use Amazon Web Services' Elastic Load Balancer or CloudFront services, which are used by multiple clients, if Amazon had to dedicate a separate IP address for each, it would break their business model1.

Sometimes, those daemons will not use SNI, which means they will get your default server. In my case, it’s password-protected with a different password than the CalDAV one, which is what triggers the “enter password” dialog. At other times, they will call your CalDAV server with dubious URLs like /.well-known/caldav, /principals/, /dav/principals/, /caldav/v2 and if your server has a different HTTP password for that and sends back a HTTP 401 status code instead of a 404 Not Found, well, that will also trigger a reauthentication prompt.

Big Sur running on my M1 MacBook Air seems to be more consistent about using SNI, but will still poke around on those URLs, triggering the reauthentication prompts.

In other words, the only way to get and Apple-compatible calendar server running reliably is to dedicate an IP and port to it that is not shared with anything else. I only have one IP address at home where the server runs, and I run other vital services behind HTTPS, so I can’t dedicate 443 to a CalDAV server. Fortunately, the configuration will accept the syntax example.org:8443 to use a non-standard port (make sure you use the Advanced option, not Automatic), but this is incredibly sloppy of Apple.


  1. Amazon does in fact have a Legacy Clients Support option, but they charge a $600/month fee for that, and if you need more than two, they will demand written justification before approving your request. ↩︎

Edgewalker, a DIY VPN server

TL:DR Don’t trust VPN services, roll your own with this easy script.

Rationale

There are many reasons to use a Virtual Private Network. Perhaps you are on an unsecured WiFi network. Perhaps you don’t want your Internet Service Provider to snoop on your browsing history using Deep Packet Inspection and compile a marketing dossier on your. Perhaps like my daughter you want to access video content on Netflix that is not available in your country. Perhaps you want to bypass the nanny state content filters the British government mandates.

Most VPN services are untrustworthy. You depend on the VPN provider’s assurances to protect your privacy, which completely defeats the purpose of a VPN. The only way you can be sure is to run your own, but baroque network protocols engendering complex software makes it difficult to do so even for the technically savvy.

Streisand was one of the first efforts to automate the process, using cloud virtual servers as the hosts operating the VPN. Trail of Bits implemented Algo to simplify it and remove some questionable choices Streisand made (although, to be fair, the Streisand project seems to have jettisoned many of them and converged on WireGuard).

Edgewalker is similar, but awesomer:

  • It is based on OpenBSD, widely considered the most secure general-purpose OS, rather than Linux.
  • Like Algo, it implements IPsec/IKEv2/MOBIKE rather than OpenVPN (read the Algo announcement for the reasons why).
    • IPsec/IKEv2 works out of the box on iOS, iPadOS and macOS.
    • In theory on Windows as well, although I have no idea how to make it work or simplify setup, any help is welcome.
  • It also implements WireGuard (recommended for Linux and Android, along with travel VPN-capable routers like the GL.iNet Mango)
  • It uses QR codes to simplify installation as much as possible on the client devices.
  • It uses Let’s Encrypt so your IPsec certificates just work (WireGuard does not rely on PKI)
  • It uses its own Unbound DNS server with DNSSEC validation support, for better privacy
  • It has no dependencies on Ansible, Python or anything else exotic you need to add on your own machine, other than a SSH client.
  • It is just a shell script with little bits of Python thrown in like Acme-Tiny, and easily auditable.

While you can run the script again as your Let’s Encrypt certificates expire (although it generates new credentials each time), I recommend simply destroying the VM and creating a new one. Of course, if you are running on physical hardware, you will want to rerun the script. If using WireGuard only, you don’t need to rerun the script as WireGuard keys do not expire and there are no certificates.

Prerequisites

You need:

  • A Let’s Encrypt account and key (I’m working on setting this up automatically for you, in the meantime you can use Step 1 on this page to do that for you).
  • An OpenBSD machine reachable from the Internet (it can be a physical machine you own, or a cloud VM like Vultr).
  • The ability to add a DNS record for the machine’s IP address (IPv4 only for now).
  • The 80x25 OpenBSD console does not support UTF-8 and cannot display the QR code in a single screen. Use a different terminal, or enter the profile URL by hand.

If you have a firewall in front of the OpenBSD machine, it needs to allow the following inbound traffic (possibly using static port mappings if you use NAT):

  • SSH (TCP port 22) so you can actually log in to your machine.
  • HTTP (TCP port 80) and HTTPS (TCP port 443) to allow Let’s Encrypt certificate issual and allow you to get the Apple-format Profiles that will ease setup on your iDevice.
  • UDP ports 500 (IKE), 1701 (IPsec) and 4500 (IPsec NAT traversal).
  • Optionally IPsec protocols ESP (IP protocol number 50, hex 0x32)) and AH (decimal 51 hex 0x33) and ESP for maximum efficiency, although many firewalls won’t support this.
  • UDP port 51820 (WireGuard).

Instructions

  • Clone the Github repository into one of your own, or copy the file edgewalker.sh somewhere you can download it without it being tampered with in transit, in practice that means HTTPS.
  • Edit the first lines in the script edggewalker.sh (X509 and USERNAME). Not strictly necessary, but make it your own.
  • Log in as root on your OpenBSD machine, then:
    pkg_add wget
    wget -c https://raw.githubusercontent.com/YOUR_GITHUB_ACCOUNT_HERE/edgewalker/main/edgewalker.sh
    sh -e edgewalker.sh
    
  • The script will ask you for:
    • The DNS name of your OpenBSD machine.
    • To copy-paste your Let’s Encrypt account key in PEM format.
  • It will then obtain Let’s Encrypt certificates, generate a QR code that you can use to download the profile on your iDevice to set up the VPN.

Credits

  • The OpenBSD team, for making their wonderful security-focused OS.
  • Reyk Flöter for making OpenIKEd, a breath of fresh air in the unnecessarily convoluted world of VPN software.
  • Jason A. Donenfeld for inventing WireGuard.
  • Let’s Encrypt, for making certificates cheap and easy.
  • Daniel Roesler for the fantastic Acme-Tiny.

Demo

I created a fresh OpenBSD 6.8 VM vpn42.majid.org on Vultr, and here is what the experience looks like:

Here is how to install the VPN on an iPhone:

Here is how to create a suitable VM on Vultr:

Canon Powershot Zoom review

Exporting secrets from the Lockdown 2FA app

I am (very) slowly migrating away from the Mac to Ubuntu Linux as my main desktop operating system. The reasons why Apple has lost my confidence are:

  • The execrable software quality of recent releases like Catalina (I plan on sticking with Mojave until I have migrated, however long that takes).
  • Apple’s increasing locking down of macOS in ways antithetical to software freedom, e.g. SIP or the notarization requirements in Catalina with the denial of service implications
  • The fact they no longer even pretend not to price-gouge on the Mac Pro. My days of buying their professional workstations every 5 years have come to an end after 15 years (PowerMac G5, Nehalem Mac Pro, 2013 Mac Pro)
  • As the iPhone market is saturating, in their eagerness to come up with a replacement growth engine in “services”, they are pushing app developers towards the despicable and unacceptable subscription licensing model
  • The butterfly keyboard fiasco exemplifies the contempt in which the company seems to hold its most loyal customers

On the plus side, ThinkPads have decent keyboards, unlike all Apple laptops since at least 2008, the LG Gram 17 is both lightweight, powerful and its huge screen is a boon to my tired eyes, and I am favorably impressed with the deep level of hardware integration offered by Ubuntu (e.g. displaying the logo and boot status in the UEFI stages of boot), even if I am not enamored of the software bloat or systemd.

One of the tasks in my migration checklist is to find a replacement for my TOTP two-factor authentication solution, which is currently the Lockdown app on iOS, iPadOS and MacOS (based on this recommendation, not to be confused with the Lockdown firewall/VPN app). I don’t trust Authy, they have a record of security failures introduced by their attempts to extend standard TOTP with their proprietary garbage, but I digress…

Thus I need to export Lockdown secrets. The iOS app can print or email a PDF with QR codes as a backup, but that’s not a very usable format for migration.

As I had to add a new TOTP secret to the app recently, that was the impetus to do this as a weekend project. I implemented a small utility called ldexport in Go to decode Lockdown for Mac’s internal file into either JSON or HTML format. Here are some simulated samples:

[
    {
        "Service": "Amazon",
        "Login": "amazon@example.com",
        "Created": "2015-11-18T19:53:34.969532012Z",
        "Modified": "2015-11-18T19:53:34.969532012Z",
        "URL": "otpauth://totp/Amazon%3Aamazon%40example.com?secret=M7IoBWqA2WuzYG27ju82XTWsflPEha3xBafMQ3i9CgwKgp6RdBGh\u0026issuer=Amazon",
        "Favorite": true,
        "Archived": false
    },
    {
        "Service": "PayPal",
        "Login": "ebay@example.com",
        "Created": "2019-11-25T08:46:57.253684043Z",
        "Modified": "2019-11-25T08:46:57.253684043Z",
        "URL": "otpauth://totp/PayPal:ebay@example.com?secret=3gB0VWJFkaYcVIiD\u0026issuer=PayPal",
        "Favorite": false,
        "Archived": false
    },
    {
        "Service": "Reddit",
        "Login": "johndoe",
        "Created": "2020-08-07T19:58:37.930042982+01:00",
        "Modified": "2020-08-07T19:58:37.930042982+01:00",
        "URL": "otpauth://totp/Reddit:johndoe?secret=nDTxDMI6bEgVpHWCViZjDFhXKH1bysRa\u0026issuer=Reddit",
        "Favorite": true,
        "Archived": false
    },
    {
        "Service": "GitHub",
        "Login": "",
        "Created": "2016-05-04T19:04:12.495128989+01:00",
        "Modified": "2017-04-04T06:33:10.641680002+01:00",
        "URL": "otpauth://totp/github.com/johndoe?issuer=GitHub\u0026secret=bXh5qmeTMzcatKKz",
        "Favorite": false,
        "Archived": false
    },
    {
        "Service": "Google",
        "Login": "johndoe@gmail.com",
        "Created": "2015-11-13T05:06:07.103500008Z",
        "Modified": "2015-11-13T05:06:07.103500008Z",
        "URL": "otpauth://totp/Google%3Ajohndoe%40gmail.com?secret=o5MvqdWDt7ZEHHSTuH6rCAUr4M6ozGQD\u0026issuer=Google",
        "Favorite": false,
        "Archived": false
    }
]

Please update to Temboz 4.4.0 or later

TL:DR If you are using my Temboz feed reader, please update as soon as possible to version 4.4.0 or later.

This is somewhat related to the last security advisory for Temboz. Fields like article title, author or tags, or feed title or description that are supposed to be plain text (not even HTML) were not being sanitized. For XSS. The effect was demonstrated by this article in BoingBoing.

Once again I apologize for potentially exposing you to XSS attacks via malicious feeds, and I would recommend you subscribe to my RSS feed for it so you can get important announcements like this one in the future.

DNP D820A review

A very solid and trouble-free printer that makes excellent prints, including spectacular panoramics, for a significant fixed price.

Despite striving for the paperless office, and believing photographic prints are mostly a relic, I have a substantial collection of printers (as my daughter points out, it’s 5 printers per person in my household):

  • HP OfficeJet Pro X551dw (extremely fast using PageWide fixed head technology, quite economical, huge paper tray capacity, very bulky)
  • Epson EcoTank ET-16600 (prints and scans A3, very economical, also very bulky but not considering the print size)
  • Brother QL-700, QL-820NWB, QL-1110NWB label printers (can make labels any length you want, the latter two are AirPrint compatible)
  • Rollo label printer (will take practically any label stock you can throw at it)
  • Fuji Instax SP100 instant photo printer (kids love them)
  • Canon Selphy QX10 portable dye-sub sticker printer for my daughter
  • two Dai-Nippon Printing DNP DS820A 8" dye-sub printers, one in storage
  • An Epson Stylus Photo R2400 in storage
  • a couple of Brother TZe label makers
  • a Dymo LabelWriter 450 Twin Turbo (unreliable garbage, at least on Mac, avoid)
  • A Selpic P1 on the way
  • A Prusa i3 MK3S 3D printer (not sure if that counts)

The DNP DS820A replaced my Epson R2400 for two reasons:

  • I print seldom enough that inks clogging in the nozzles was a big issue.
  • The Epson is a behemoth that is very hard to find a place for, even before I downsized.

The DNP uses dye-sublimation technology to make its prints. You may have encountered one at a drugstore self-service photo kiosk, or at photo events like Macy’s Santa Claus portrait sessions. These printers are designed specifically for these two use cases, and are built like tanks with a steel chassis. Since most events typically gang two or even four printers to maximize throughput, they are also very compact, with a footprint barely larger than an A3 sheet of paper, mine is on a lower shelf in my IKEA FREDDE computer desk.

Until the advent of fine-art photo printers with 6 or more color pigment inks, dye-sub was the top-end digital photo printing technology, thanks to the continous tones it can generate, like photographic processes (e.g. Fuji Frontier or Noritsu QSS digital minilabs, or large-format laser enlargers like the Cymbolic Sciences LightJet or Durst Lambda/Theta). Dye-subs have all but disappeared from the consumer market, however, apart from some Canon Selphy compact printers, and are now largely reserved for professional applications, with a price to match. The DNP DS820A used to cost $1100. They lowered the price to under $1000 a few years ago, but cheaped out by removing the print-catching basket that used to be included in the older package.

You pop off the front panel and install a roll of paper and a reel of dye ribbons in a tray above the paper inside the printer, then pop the front back in. Nothing protrudes and the media is protected from dust, which is really nice. There are two different sizes of media, 8x10 (130 prints) and 8x12 (110 prints). The size is mostly relevant for the dye ribbons that have CMY sections sized in increments of 10 or 12 inches, but a surprising consequence of this is that you cannot switch from 8x10 to 8x12 and vice versa (you can make smaller divisions and the printer will trim them to size using its built-in cutter). The cost per print is about $0.65 for 8x10, $0.80 for 8x12, $1.30 if you get the premium metallic paper. Since the paper and ribbon is consumed no matter the coverage, it’s a constant, unlike the variable costs of an inkjet printer.

The print quality is excellent, as can be expected, as is the color calibration out of the box. It may not quite have the tonal subtlety of an Epson, but there is no visible pixellation. Furthermore, the prints get a clear protective laminate, which makes them smudge-proof and very tough. You can even choose one of four different finishes applied by a roller so no media change required: glossy, luster, matte and fine matte.

One of the marquee features of the DS820A and its little 6" brother the DS620A is the ability to make panoramic prints. Each print is made by combining multiple pages together, with about 2" of overlap wastage, so if your printer is loaded with 8x12 media you can make 8x22 or 8x32 prints, with 8x10 media you can make 8x18 or 8x26. The 8x32 panoramic prints are absolutely spectacular, although finding a suitable frame for them is not a trivial undertaking, that not being a standard print size.

Unfortunately this functionality is not built into the printer driver, but you must use the DNP Hot Folder utility, and while it is available for both Mac and Windows, only the Windows version can make panoramic prints. DNP Hot Folder is meant to use for events where a single PC or server controls multiple printers. You drop the files into a directory per print size (hence the name “Hot Folder”) and the software will automatically print it on the next available printer loaded with the right media. Since the printers run in parallel, even if the print speed is not incredibly fast (about 30 to 60 seconds per print), aggregate throughput is sufficient for a busy event. I have mine on a USB switch (the printer has no network connectivity) to share it between my Mac and my gaming PC.

Critical Temboz vulnerability, please update

TL:DR If you are using my Temboz feed reader, please update as soon as possible to version 4.0 or later.

This is because Temboz depends on feedparser, Kurt McKee’s (originally Mark Pilgrim’s) ultra-liberal feed parsing library for Python. One of its responsibilities is to sanitize feed content to strip out potentially dangerous HTML like <script> tags. Unfortunately, I only just realized that on Python 3, due to the absence of the sgmllib module in Python 3 that used to ship with Python 2, feedparser will silently fail and not sanitize the HTML instead of failing safe, e.g. throwing a NotImplementedError. Since this is such a fundamentally flawed approach, I decided no longer to trust feedparser withis responsibility and assign it to Mozilla’s bleach instead. Furthermore, Temboz will now perform a sanity check at startup and refuse to start if <script> tags are not being filtered.

I apologize for potentially exposing you to XSS attacks via malicious feeds. Unfortunately I have no way to reach out to all those who installed Temboz. If you are installing Temboz, I would recommend you subscribe to my RSS feed for it so you can get important announcements like this one in the future.

Making ScanSnap Receipts usable

For a long time I used a service called Shoeboxed to scan and organize my credit card receipts. Basically stuff your receipts in a US prepaid envelope, drop it in the mail, and they scan, OCR and shred them, as well as analyzing the text to extract the information. Unfortunately, since I moved to the UK the service leaves to be desired, and the price has also gone up over time.

I have a couple of Fujitsu ScanSnap document scanners, a S1500M, which is no longer supported on macOS Mojave (but fortunately is by the third-party app ExactScan), and an iX100 which still is supported, as well as a SV600 which is utterly unsuited to dealing with crumpled receipts. The new, dumbed-down ScanSnap Home app that ships with ScanSnaps has a receipt mode. Since I never use the iX100 to scan documents at home given I have a S1500M (it’s a handheld battery=powered simplex scanner that’s mostly intended for mobile use), I dedicated it and ScanSnap Home to scanning receipts.

The basic functionality of scanning, deskewing, OCR-ing and extracting date, amount, vendor and so on mostly works, but otherwise ScanSnap Receipts is an ergonomic disaster. For starters, it never recognizes the currency correctly and always identifies my transactions as being in dollars rather than pounds. Secondly, it inexplicably lacks the ability to batch-edit receipts, e.g. select the date range for my last trip to France and change all of them from dollars to euros. You need to edit them one by one, which is as incredibly tedious as you can imagine.

After a few weeks of this, I decided to take matters in my own hand. It turns out ScanSnap Home uses Core Data backed by an underlying SQLite database. SQLite is the world’s most widely deployed database (every single Android and iOS smartphone includes it, for starters), but the Core Data object-relational mapper above it does a terrific job of obfuscating it and reducing its performance. Nonetheless, after a little bit of digging, I wrote the following script to automate the most repetitive operations:

  1. Stop all ScanSnap auxiliary processes, as they have the DB opened even if you quit the app
  2. Set the currency for all transactions tagged as Unchecked (ScanSnap Home does this by default) to GBP
  3. Normalize all unchecked Waitrose vendor names to Waitrose
  4. Same for Tesco
  5. Rename M&S to Marks & Spencer
  6. Fixes for Superdrug and Sainsbury’s
  7. Set the amount of receipts tagged as duplicates to zero
  8. Attempt to fix little-endian European format DD/MM/YYYY dates parsed as middle-endian US format MM/DD/YYYY
  9. Attempt to fix Euro format DD/MM/YY dates parsed as US format YY/MM/DD
  10. Restart ScanSnap Home

The timestamps in the database are in a strange format that seems to be the number of seconds since an epoch of 2001-01-01T00:00:00 UTC. I hope no one tries to scan receipts older than that… You can convert to UNIX timestamps by adding 978307200 and from there to SQLite’s Julian Day format.

One major annoyance thing this script attempts to fix is dates. Because date formats are ambiguous (is 11/3/20 March 11 2020, or November 3 2020 or perversely March 20 2011?) and point-of-sale vendors are neither ISO 8601 nor even Y2100 compliant, parsing dates is a minefield. My assumption is that receipts will be scanned in a reasonably timely manner, and if there is ambiguity, the closest date should win.

#!/bin/sh
pkill -9 -f ScanSnap

sqlite3 "$HOME/Library/Application Support/PFU/ScanSnap Home/Managed/ScanSnapHome.sqlite" << EOF

.mode lines

UPDATE zcontent
SET zcurrencysign=(SELECT z_pk FROM zcurrencysign WHERE zvalue='GBP')
WHERE zdoctype=4 AND z_pk IN (
  SELECT z_4contents
  FROM z_4labels
  JOIN zlabel ON z_15labels=zlabel.z_pk
  WHERE zlabel.zname='Unchecked'
);

UPDATE zcontent
SET zvendor=(SELECT z_pk FROM zvendor WHERE zvalue='Waitrose')
WHERE zdoctype=4 AND z_pk IN (
  SELECT z_4contents
  FROM z_4labels
  JOIN zlabel ON z_15labels=zlabel.z_pk
  WHERE zlabel.zname='Unchecked'
) AND zvendor IN (
  SELECT z_pk FROM zvendor
  WHERE zvalue<>'Waitrose' AND zvalue LIKE '%waitrose%'
);

UPDATE zcontent
SET zvendor=(SELECT z_pk FROM zvendor WHERE zvalue='Tesco')
WHERE zdoctype=4 AND z_pk IN (
  SELECT z_4contents
  FROM z_4labels
  JOIN zlabel ON z_15labels=zlabel.z_pk
  WHERE zlabel.zname='Unchecked'
) AND zvendor IN (
  SELECT z_pk FROM zvendor
  WHERE zvalue<>'Tesco' AND zvalue LIKE '%tesco%'
);

UPDATE zcontent
SET zvendor=(SELECT z_pk FROM zvendor WHERE zvalue='Marks & Spencer')
WHERE zdoctype=4 AND z_pk IN (
  SELECT z_4contents
  FROM z_4labels
  JOIN zlabel ON z_15labels=zlabel.z_pk
  WHERE zlabel.zname='Unchecked'
) AND zvendor IN (
  SELECT z_pk FROM zvendor
  WHERE zvalue LIKE '%M&S%'
);

UPDATE zcontent
SET zvendor=(SELECT z_pk FROM zvendor WHERE zvalue='Superdrug')
WHERE zdoctype=4 AND z_pk IN (
  SELECT z_4contents
  FROM z_4labels
  JOIN zlabel ON z_15labels=zlabel.z_pk
  WHERE zlabel.zname='Unchecked'
) AND zvendor IN (
  SELECT z_pk FROM zvendor
  WHERE zvalue<>'Superdrug' AND zvalue LIKE '%superdrug%'
);

UPDATE zcontent
SET zvendor=(SELECT z_pk FROM zvendor WHERE zvalue='Sainsbury''s')
WHERE zdoctype=4 AND z_pk IN (
  SELECT z_4contents
  FROM z_4labels
  JOIN zlabel ON z_15labels=zlabel.z_pk
  WHERE zlabel.zname='Unchecked'
) AND zvendor IN (
  SELECT z_pk FROM zvendor
  WHERE zvalue<>'Sainsbury''s' AND zvalue LIKE '%Sainsbury%'
);

UPDATE zcontent
SET zamount=0.0
WHERE zdoctype=4 AND z_pk IN (
  SELECT z_4contents
  FROM z_4labels
  JOIN zlabel ON z_15labels=zlabel.z_pk
  WHERE zlabel.zname='Duplicate'
);

UPDATE zcontent
SET zreceiptdate = strftime('%s', 
  strftime('%Y-%d-%m', zreceiptdate+978307200, 'unixepoch', 'localtime'),
  'utc'
)-978307200
WHERE zdoctype=4 AND z_pk IN (
  SELECT z_4contents
  FROM z_4labels
  JOIN zlabel ON z_15labels=zlabel.z_pk
  WHERE zlabel.zname='Unchecked'
)
AND zreceiptdate IS NOT NULL
AND strftime('%s', 
  strftime('%Y-%d-%m', zreceiptdate+978307200, 'unixepoch', 'localtime'),
  'utc'
)-978307200
BETWEEN zreceiptdate AND strftime('%s', 'now')-978307200;

UPDATE zcontent
SET zreceiptdate = strftime('%s',
  strftime('20%d-%m-', zreceiptdate+978307200,
           'unixepoch', 'localtime') ||
  substr(strftime('%Y', zreceiptdate+978307200,
         'unixepoch', 'localtime'), 3),
  'utc'
)-978307200
WHERE zdoctype=4 AND z_pk IN (
  SELECT z_4contents
  FROM z_4labels
  JOIN zlabel ON z_15labels=zlabel.z_pk
  WHERE zlabel.zname='Unchecked'
)
AND zreceiptdate IS NOT NULL
AND strftime('%s',
  strftime('20%d-%m-', zreceiptdate+978307200,
           'unixepoch', 'localtime') ||
  substr(strftime('%Y', zreceiptdate+978307200,
         'unixepoch', 'localtime'), 3),
  'utc'
)-978307200
BETWEEN zreceiptdate AND strftime('%s', 'now')-978307200;

EOF

open /Applications/ScanSnapHomeMain.app

Note that the SQLite database is not used only by expenses (ZCONTENT.ZDOCTYPE=4) but also to store a summary of all documents scanned. Also, the ZCONTENT table has a column ZUNCHECKED that is not what you would expect, it is a constant 1 even if you remove the Unchecked tag from the transaction.

Now, all the usual disclaimers apply, modifying the database directly is not something supported by the app developer, and could have unintended consequences. If you use this script (or more likely modify it for your needs), I disclaim responsibility for any damages or data loss this may cause.

Update (2020-10-01):

Added:

  • setting duplicate receipt amounts to 0
  • fixing DD/MM/YYYY dates misparsed as MM/DD/YYYY
  • fixing DD/MM/YY dates misparsed as YY/MM/DD

Virtual Reality for the people

I have been shooting stitched panoramas for almost 20 years. I have used manual panorama heads like the Kaidan Kiwi+ and more recently the pocketPANO Compact, robotic heads like the Gigapan EPIC 100, and four successive generations of the Ricoh Theta (Theta, Theta S, Theta V, Theta Z1).

Setting up and iterating a manual head is incredibly tedious. The Gigapan makes it less so, specially when using long lenses (my standard setup is a Leica M typ 240 or M10 with a 90mm f/2 Apo-Summicron-M ASPH). The Theta series was a major breakthrough in that it could produce nearly seamless 360° panoramas with no motion artifacts or ghosting. The Z1 with its large 1″ sensor finally yields image quality that I am happy with.

The viewing situation has also improved. In the early days you needed Java applets or dubious plug-ins. Nowadays, it can all be done in HTML5 with the aid of JavaScript libraries like Panellum. The user experience is still one of scrolling an image through a rectangular viewport in browser window. The experience on mobile is a bit better because it can use the accelerometer so you scroll by panning with your phone or tablet. It’s still not a fully immersive experience.

This Friday Facebook announced a price drop for its Oculus Go VR headset, the entry-level 32GB model being at a near-impulse purchase price of $150, and of course I yielded to the impulse. I had bought the original Oculus Rift to get a sense of what the potential of VR was, but tethered to a beefy PC, it made for impressive demos but not much more.

The Oculus Go changes this completely because it is standalone (it has the guts of a midrange smartphone circa 2018) and affordable. One of the ways they kept costs down is by removing motion tracking: it can detect angular motions of your head, but not when you are walking around, but for purposes of viewing 360° panoramic stills and videos, that is not required.

One of my concerns was how deeply it would be tied to the Facebook privacy-mangling machine. My New Year’s resolution for 2019 was to delete my FB account (my 2020 resolution was to switch all my digital camera clocks to UTC and never again bother with the abomination that is Daylight Saving Time)—underpromise and overdeliver, that’s my motto… Any requirement to have a FB account would be a total deal-breaker for me.

The second concern was how much of a hassle it would be to set up and use with my own photos. Camera-makers are not known for outstanding software and Ricoh is no exception. There is an Oculus third-party app for Theta cameras, but it hasn’t been updated in ages and only lists Theta S compatibility.

I was pleasantly surprised at how smoothly it went. You can avoid the FB account by using an Oculus account (I used mine from the Rift), and no additional apps are required. Just install the Android File Transfer utility if you are on a Mac, copy the files to the headset’s Pictures directory. I would recommend using subfolders because the built-in Gallery app is not smart about caching thumbnails and is very slow at regenerating the view if there are more than about 20 images or so in a folder.

The image quality is not exceptional. Mike Abrash, who worked on the ground-breaking 3D game Quake, and is now Chief Scientist at Oculus, says fully immersive VR requires resolution halfway between 4K and 8K in each eye (vs. 2.5K shared for both eyes on the Go), and is at least a decade away. The immersive nature of the Go does provide that elusive Wow! factor, however, and more than makes up for its designed-to-a-budget shortcomings. The 2560×1440 display with an apparent field of view of 100° yields 3.7MP in the FOV but spherical trigonometry calculations reveal the entire 360° sphere would require a 26MP image to cover it entirely, which is slightly more than the 23MP images the Theta Z1 delivers. Fully immersive VR requires very high resolutions!

It even handles video transparently (you do have to convert Theta videos from the native format to equirectangular projection video with Ricoh’s app, which is excruciatingly slow). Keep in mind that video sizes are large, and with a 32GB model, there are limits to how much you can store on the device. If you plan to view immersive videos, the 64GB model is highly recommended.

The Oculus Go also has a “Cast” feature that will stream what the person wearing the headset is seeing to the phone it is paired with. You can have a friend wear the headset and narrate what they are seeing, I tried this with my architect mother-in-law as I was showing her the sights in Jerusalem, much to her delight (her master’s thesis at SOAS was on the Dome of the Rock). The Go has a unique sound projector developed by Oculus that means the user doesn’t have to wear earbuds, and can hear you speak. I would recommend you change the default display sleep time from the ridiculously short 15 seconds to 3 to 5 minutes, so you can swap the headeet without losing the cast session or resetting the app. Sadly, the battery life is nothing to write home about. I would guesstimate it at 1 to 2 hours, tops.

I still need to figure how to share my 360° VR photos using WebVR so other people can view them from their own Oculus Go (or other headsets).

One essential accessory for the Z1 or another similar 360° camera is a selfie stick or similar implement, otherwise your hands will appear prominently in the final panorama. Ricoh sells three models.

The TM-1 is a very well designed tripod (rumored to be made by Velbon) with a magnetic quick-release mount. It’s easy to deploy with one click, unlike a conventional tripod, and fully extended the camera is at eye height for a natural perspective.

The TM-3 is a short telescopic stick. It’s long enough that your hands no longer appear in the picture but low-profile enoough that the TM-3 itself is invisible. It is well-made, unlike most generic Chinese selfie sticks, unlocks and locks with a simple twist, and the TS-2 case for the Z1 has an opening at the bottom so you don’t need to detach it before putting the camera back in its case, a nice touch.

The TM-2 is a longer version of the TM-3 with an unnecessary swivel head, I haven’t tried it but the swivel head would defeat the invisible factor.

The battle of the Tech Dopp kits

Most travelers pack toiletry bags, known in the US as Dopp kits, to organize their toiletries and prevent them from leaking into their luggage. We need much the same to hold the vast assortment of chargers, cables, dongles and other technological paraphernalia required to function in this day and age. Sadly, the state of quality design in tech dopp kits is sorely lacking at the moment.

My daily loadout includes:

When traveling I add:

  • Apple USB Type-C Digital AV Multiport Adapter (the latest version with 4Kp60 and HDR support)
  • Onkyo DP-X1 high-resolution audio player
  • 4-port Anker USB charger
  • 3 USB-A to Lightning cables (one for my iPhone, one for my iPad, one for my wife). The USB-A connectors add a surprising amount of bulk, I wish Anker would make a charger with some sort of built-in octopus cable and cable management.
  • 2 USB-A to micro-USB cables
  • 20,000 mAh Mophie Powerstation XXL USB-C Power Delivery battery pack (can be used to charge the MacBook or iPad Pro or expand their battery life). Much lighter and convenient than the Anker 26,800 mAh lead ingot it replaced.
  • Ricoh Theta Z1 360° panoramic camera and TM-3 stick
  • Novoflex Mikrostativ mini-tripod
  • spare batteries for whatever camera I packed
  • Watson travel camera battery charger and whatever interchangeable battery plates are needed
  • Samsung T3 256GB USB SSD
  • a 18650 flashlight (BLF FW3A or Zebralight SC600FcIV+) and Olight USB-powered 18650 battery charger

Muji hanging toiletry kit

I used this around 2008 when I got the first-generation MacBook Air, to hold its charger and the Sanho Hyperdrive, an early battery pack that had a MagSafe connector until Apple sicced their lawyers at them. It has room for the two and the video dongles, not much else.

Waterfield Design gear pouches

Waterfield Designs was one of the first companies to make iPod cases, and they later added a larger pouch to accommodate an iPod and a bunch of other accessories. They have padded neoprene pockets and are very well made, but the padding adds a tremenoud amount of bulks, and they are also quite large.

Unfortunately they seem to have discontinued these pouches and replaced them with new ones in waxed canvas and leather that might arguably look a little better, but are less practical.

Bond Travel Gear EDC organizer Escapade gear pouch

On paper this is an interesting option. Unfortunately it is quite stiff, which makes it hard to pack, and overengineered with too many small pockets that reduce the effective carrying capacity because there is too much stitching, webbing, zippers, padding and other organizational overhead for the actual contents.

Eagle Creek Etools Organizer Pro

This bag is quite large, it can hold a 10.5″ iPad Pro. The fabric is thin and pliant, and the pockets large. This is a much better option for me as I can put larger items like battery packs, calculators, music players and so on. If it were ever slightly larger, it would also be a great bag to carry necessities on a flight so you don’t have to rummage for them in the overhead bin.

Update (2019-10-17):

It seems this pouch has been discontinued, but it is still readily available as new-old-stock with the usual retailers.

Triple Aught Design OP10

This tacticool pouch replete with mil-spec MOLLE attachments is better designed than the Bond, but still too small and with too many pockets and loops to maximize utility. It’s best used for those who want to carry an assortment of long thin objects like knives, tools, flashlights or cables. I would guesstimate the number of mall ninjas to operators owning this pouch is 10:1.

Incase Nylon accessory organizer

This organizer is recommended by The Wirecutter in their 2019 roundup. The fabric is thin and more flexible than overengineered ones in Cordura or ballistic nylon, which helps keep bulk down and increases the pockets' carrying capacity. The assortment of pockets is very well designed, at least for my needs, and it’s near ideal for a flight (it won’t hold an iPad, though).

Conclusions

The Eagle Creek is the best option. It doesn’t add too much weight and maximizes carrying capacity to space. No one has yet solved the problem of organizing cables so they don’t devolve into a gordian knot, however.

My top 100 things to eat in San Francisco

Incomplete,never finished, and probably completely out of date after the aftermath of coronavirus. Check out my foodie map instead (although most likely just as out of date).

  1. The caramelized onion and sesame baguette at Noe Valley Bakery
  2. The chocolate-cherry breakfast bread at Noe Valley Bakery
  3. The combination seafood salad at Swan’s Oyster Bar
  4. The clam chowder at Ferry Plaza Seafood
  5. The mushroom pizza at Delarosa
  6. The salted caramel ice cream at Bi-Rite Creamery
  7. The “crunchy sticks” (sesame-poppyseed-cheese twists) at Esther’s German Bakery (available at Rainbow Grocery and on Thursdays at the Crocker Galleria)
  8. The El Rey chocolate toffee semifreddo at Bix (seasonal)
  9. The pistachio macaroons at Boulette’s Larder
  10. The burnt caramel and chocolate covered hazelnuts at Michael Recchiuti
  11. The chocolate feuilletine cake at Miette
  12. The belly buster burger at Mo’s Grille
  13. The hazelnut hot chocolate at Christopher Elbow
  14. The princess cake at Schubert’s Bakery
  15. The ricciarellis (soft bitter-almond macaroons) at Arizmendi
  16. The Charlemagne chocolate feuilletine cake at Charles Chocolates
  17. The chocolate hazelnut tarts at Tartine
  18. The croissant at Tartine
  19. The caramelized hazelnut financier at Craftsman and Wolves
  20. The bordeaux cherry ice cream at Swensen’s, in a cone and dipped in chocolate (the ice cream itself, not the cone)
  21. The flatbread at Universal Cafe
  22. The Umami truffle burger
  23. Captain Mike’s smoked tuna (Ferry Plaza farmer’s market)
  24. The mushroom empañadas at El Porteño
  25. The Tcho chocolate liquid nitrogen ice cream at Smitten
  26. The macarons at Chantal Guillon
  27. Pine nut Bacetti (Howard & 9th)
  28. The Atomica pizza at Gialina’s
  29. The salted hazelnut and chocolate shortbread at Batter Bakery
  30. The chocolate velvet cupcake at Kara’s Cupcakes
  31. The lobster roll at Woodhouse Fish Co.
  32. The gianduja and pistachio at Coletta Gelato
  33. The croissants and pains au chocolat at Arsicault (Arguello & Clement)
  34. The lasagna served bubbling hot from the oven at Pazzia
  35. The wild mushroom benedict at Mission Beach Cafe
  36. The farm egg ravioli at Cotogna
  37. The cioppino (spicy tomato-fish stew) at The Tadich Grill
  38. The sada (plain) dosa at Dosa Fillmore
  39. The French and Fries burger at Roam Artisan Burgers
  40. The lasagna at Trattoria di Vittorio

Migrating to Hugo

I have been meaning to move away from Wordpress to a static site generator for a very long time, due to:

  • The slowness of WP, since every page request makes multiple database calls due to the spaghetti code nature of WP and its plugin architecture. Caching can help somewhat, but it has brittle edge cases.
  • Its record of security holes. I mitigated this somewhat by isolating PHP as much as possible.
  • It is almost impossible to follow front-end optimization best-practices like minimizing the number of CSS and JS files because each WP plugin has its own

My original plan was to go with Acrylamid, but about a year ago I started experimenting with Hugo. Hugo is blazing fast because it is implemented in Go rather than a slow language like Python or Ruby, and this is game-changing. Nonetheless, it took me over a year to migrate. This post is about the issues I encountered and the workflow I adopted to make it work.

Wordpress content migration

There is a migration tool, but it is far from perfect despite the author’s best efforts, mostly because of the baroque nature of Wordpress itself when combined with plugins and an old site that used several generations of image gallery technology.

Unfortunately, that required rewriting many posts, specially those with photos or embedded code.

Photo galleries

Hugo does not (yet) support image galleries natively. I started looking at the HugoPhotoSwipe project, but got frustrated by bugs in its home-grown YAML parser that broke round-trip editing, and made it very difficult to get galleries with text before and after the gallery proper. The Python-based smartcrop for thumbnails is also excruciatingly slow.

I wrote hugopix to address this. It uses a simpler one-way index file generation method, and the much faster Go smartcrop implementation by Artyom Pervukhin.

Broken asset references

Posts with photo galleries were particularly broken, due to WP’s insistence on replacing photos with links to image pages. I wrote a tool to help me find broken images and other assets, and organize them in a more rational way (e.g. not have PDFs or source code samples be put in static/images).

It also has a mode to identify unused assets, e.g. 1.5GB of images that no longer belong in the hugo tree as their galleries are moving elsewhere.

Password-protected galleries

I used to have galleries of family events on my site, until an incident where some Dutch forum started linking to one of my cousin’s wedding photos and making fun of her. At that point I put a pointed error message for that referrer and controlled access using WP’s protected feature. That said, private family photos do not belong on a public blog and I have other dedicated password-protected galleries with Lightroom integration that make more sense for that use case, so I just removed them from the blog, shaving off 1.5GB of disk in the bargain.

There are systems that can provide search without any server component, e.g. the JavaScript-based search in Sphinx, and I looked at some of the options referenced by the Hugo documentation like the Bleve-based hugoidx but the poor documentation gave me pause, and I’d rather not run Node.js on my server as needed by hugo-lunr.

Having recently implemented full-text search in Temboz using SQLite’s FTS5 extension, I felt more comfortable building my own search server in Go. Because Hugo and fts5index share the same Go template language, this makes a seamless integration in the site’s navigation and page structure easy.

Theme

There is no avoiding this, moving to a new blogging system requires a rewrite of a new theme if you do not want to go with a canned theme. Fortunately, Hugo’s theme system is sane, unlike Wordpress', because it does not have to rely on callbacks and hooks as much as with WP plugins.

One pet peeve of mine is when sites change platform with new GUIDs or permalinks in the RSS feeds, causing a flood of old-new articles to appear in my feed reader. Since I believe in showing respect to my readers, I had to avoid this at all costs, and also put in place redirects as needed to avoid 404s for the few pages that did change permalinks (mostly image galleries).

Doing so required copying the embedded RSS template and changing:

<guid>{{ .Permalink }}</guid>

to:

<guid isPermaLink="false">{{ .Params.rss_guid | default .Permalink }}</guid>

The next step was to add rss_guid to the front matter of the last 10 articles in my legacy RSS feed.

How big can a panorama get?

I use the Kolor AutoPano Giga panorama-stitching software, recently acquired by GoPro, but I have yet to produce a gigapixel panorama like those they pioneered. This brings up an interesting question: given a camera and lens, what would the pixel size of the largest 360° stitched panorama be?

Wikipedia to the rescue: using the formula for the solid angle of a pyramid, the full panorama size of a camera with m megapixels on a sensor of a x b using a focal length of f would be:

m * π / arctan(ab / 2f / sqrt(4f2 + a2 + b2))

For single-strip panoramas of height h (usually a or b), the formula would be:

m * π * h / 2f / arctan(ab / 2f / sqrt(4f2 + a2 + b2))

(this applies only to rectilinear lenses, not fisheyes or other exotics).

Here is a little JavaScript calculator to apply the formula (defaults are for the Sony RX1RII, the highest resolution camera I own):

MP
mm actual 35mm equivalent

MP
MP
MP

The only way I can break through the gigapixel barrier with a prime lens is using my 24MP APS-C Fuji X-T2 with a 90mm lens.

Update (2020-01-21):

Now I could reach 10GP with my Nikon Z7 and the Nikkor 500mm f/5.6 PF.

Scanner group test

TL:DR Avoid scanners with Contact Image Sensors if you care at all about color fidelity.

Vermeer it is not

After my abortive trial of the Colortrac SmartLF Scan, I did a comparative test of scanning one of my daughter’s A3-sized drawings on a number of scanners I had handy.

Scanner Sensor Scan
Colortrac SmartLF Scan CIS ScanLF.jpg
Epson Perfection Photo V500 Photo (manually stitched) CCD Epson_V500.jpeg
Epson Perfection V19 (manually stitched) CIS Epson_V19.jpg
Fujitsu ScanSnap S1500M (using a carrier sheet and the built-in stitching) CCD S1500M_carriersheet.jpg
Fujitsu ScanSnap SV600 CCD SV600.jpg
Fuji X-Pro2 with XF 35mm f/1.4 lens, mounted on a Kaiser RS2 XA copy stand with IKEA KVART 3-spot floor lamp (CCT 2800K, a mediocre 82 CRI as measured with my UPRtek CV600) CMOS X-Pro2.jpg

I was shocked by the wide variance in the results, as was my wife. This is most obvious in the orange flower on the right.

Comparison

I scanned a swatch of the orange using a Nix Pro Color Sensor (it’s the orange square in the upper right corner of each scan in the comparison above). When viewed on my freshly calibrated NEC PA302W SpectraView II monitor, the Epson V500 scan is closest, followed by the ScanSnap SV600.

The two scanners using Contact Image Sensor (CIS) technology yielded dismal results. CIS are used in low-end scanners, and they have the benefit of low power usage, which is why the only USB bus-powered scanners available are all CIS models. CIS sensors begat the CMOS sensors used by the vast majority of digital cameras today, superseding CCDs in that application, I would not have expected such a gap in quality.

The digital camera scan was also quite disappointing. I blame the poor quality of the LEDs in the IKEA KVART three-headed lamp I used (pro tip: avoid IKEA LEDs like the plague, they are uniformly horrendous).

I was pleasantly surprised by the excellent performance of the S1500M document scanner. It is meant to be used for scanning sheaves of documents, not artwork, but Fujitsu did not skimp and used a CCD sensor element, and it shows.

Pro tip: a piece of anti-reflective Museum Glass or equivalent can help with curled originals on the ScanSnap SV600. I got mine from scraps at a framing shop. I can’t see a trace of reflections on the scan, unlike on the copy stand.

Update (2018-10-14):

Even more of a pro tip: a Japanese company named Bird Electron makes a series of accessories for the ScanSnap line, including a dust cover for the SV600 and the hilariously Engrish-named PZ-BP600 Book Repressor, essentially a sheet of 3mm anti-reflection coated acrylic with convenient carry handles. They are readily available on eBay from Japanese sellers.

Colortrac SmartLF Scan review

TL:DR summary

Pros:

  • Scans very large documents
  • Easy to use
  • Packs away in a convenient carrying case

Cons:

  • So-so color fidelity
  • Hard to feed artwork straight
  • Dust and debris can easily get on the platen, ruining scans
  • Relatively expensive for home use

Review

One thing you do not lack for when your child enters preschool is artwork. They generate prodigious amounts of it, with gusto, and they are often large format pieces on 16×24″ paper (roughly ISO A2). The question is, what do you do with the torrent?

I decided I would scan them, then file them in Ito-Ya Art Profolios, and possibly make annual photobooks for the grandparents. This brings up the logistical challenge of digitizing such large pieces. Most flatbed scanners are limited to 8.5×14″ (US Legal) format. Some like the Epson Expression 11000XL and 12000XL can scan 11×17″ (A3), as can the Fujitsu ScanSnap SV600 book scanner, but that is not fully adequate either. One option would be to fold the artwork up, scan portions then stitch them together in AutoPanoGiga or Photoshop, but that would be extremely cumbersome, specially when you have to do a couple per day. I do not have access to a color copier at my office, and most of these are only A3 anyway.

I purchased a Kaiser RS2 XA copy stand (cheaper to get it direct from Europe on eBay than from the usual suspects like B&H) and got a local framing shop to cut me a scrap of anti-reflective Museum Glass. This goes up to 16×20″ for the price of a midrange flatbed scanner, but it is tricky to set up lights so they don’t induce reflections (no AR coating is perfect), perfectly aligning the camera with the baseboard plane is difficult (I had to shim it using a cut-up credit card), and this still doesn’t solve the problem of the truly large 16×24″ artwork (stands able to handle larger formats are extremely expensive and very bulky).

I then started looking at large-format scanners like those made by Contex or Océ. They are used by architecture firms to scan blueprints and the like, but they are also extremely large, and cost $3000-5000 for entry-level models, along with onerous DRM-encumbered software that requires license dongles and more often than not will not run on a Mac. They are also quite bulky, specially if you get the optional stands.

That is why I was pleasantly surprised to learn British company Colortrac makes a model called the SmartLF Scan! (I will henceforth omit the over-the-top exclamation mark). It is self-contained (can scan to internal memory or a USB stick, although it will also work with a computer over USB or Ethernet, Windows-only, unfortunately), available in 24″ or 36″ wide versions, is very compact compared to its peers, and is even supplied with a nifty custom-fitted wheeled hard case. The price of $2,000 ($2,500 for the 36″ version), while steep for home use, is well within the range of enthusiast photo equipment. I sold a few unused cameras to release funds for one.

Once unpacked, the scanner is surprisingly light. It is quite wide, obviously, to be able to ingest a 24″ wide document (see the CD jewel case in the photo above for scale). There is a LCD control panel and a serviceable keypad-based (not touch) UI. The power supply is of the obnoxious wall-wart type. I wish they used text rather than inscrutable icons in the UI—it is much more informative and usable to see a menu entry for 400dpi resolution rather than checkerboard icons with various pitches.

After selecting your settings (or saving them as defaults), you load paper by feeding it from the front, face up. It is quite hard to feed large-format paper straight, and this is compounded by the lack of guides. On the other hand it is hard to see how Colortrac could have fitted photocopier-style guide rails in such a compact design, and they would be likely to break.

The scanner is simplex, not duplex, unsurprisingly at that price point. The sensor is on top of the feed, which helps control dust and debris sticking to it, but when scanning painted artwork, there will inevitably be crumbs of paint that will detach and stick to the sensor platen. This manifests itself as long dark vertical lines spoiling subsequent scans, something I occasionally also see on my Fujitsu ScanSnap document scanner. Cleaning the Colortrac is way easier than on the ScanSnap, as unfolding rear legs and releasing front catches opens it wide, and a few passes with optical cleaning wipes (I use Zeiss’ single-use ones) will do the trick.

By the manufacturer’s own admission, the scanner is designed to scan technical drawings, not art. It uses a linear contact image sensor (CIS) like lower-end flatbed scanners and document scanners, unlike the higher-fidelity charge-coupled device (CCD) sensors used in higher-end graphics arts and photo scanners. The light source is a row of point light LEDs that casts relatively harsh shadows on the paper. They do make CCD scanners for graphics arts, but they start at $10,000… Contex makes an A2 flatbed CCD scanner, the HD iFlex, but it costs $6,700 (at Jet.com of all places), their iQ Quattro 2490 at $4,500 is the most viable step-up (it uses a CIS, but offers 16-bit color, AdobeRGB and beyond gamut, calibration and magnetic paper guides).

The scanner’s resolution is 600dpi. Scanning 16×24″ originals at that resolution yields a 138MP file that is nearly a gigabyte in size. The 400dpi setting yields a much more reasonable 200MB or so, and compressing them further using tiffcp with zip compression (not an option on the scanner) yields 130-140MB files.

Unfortunately, I ended up returning it. There was a 1cm scratch in the glass platen, which manifested itself as streaks. It takes quite a bit to scratch glass (I don’t think it was Lexan or similar), and I wasn’t scanning sandpaper, so it must have been a factory defect or a customer return. When I looked at the color fidelity of the scans, I was not inclined to order a replacement, and got a Fujitsu ScanSnap SV600 from Japan instead from an Amazon third-party reseller (25% savings over the US price, even if you usually forgo a US warranty on grey-market imports).

Avery 22807 template for InDesign

The Avery 22807 2-inch circular stickers are a good alternative to Moo, PSPrint et al when you need a small quantity of stickers in a hurry. Unfortunately Avery has not seen it fit to provide usable InDesign templates as they do with some of their other sticker SKUs, only Microsoft Word, which is needless to say inadequate. A search for “Avery 22807 Indesign template” yielded some, but they have issues with missing linked PDF files.

I reverse-engineered the Microsoft template to build one of my own, with dimensions (including the tricky almost-but-not-quite square grid spaced at 5/8″ horizontally but 7/12″ vertically) to simplify “Step and Repeat…”.

I have only tested this with my InDesign CS6, not sure if it will work with older versions.

Avery 22807 2-inch circular labels.indt

Only the best will do for my readers

Qualys SSL Labs: A+, 100% rating

Securityheaders.io: A+ rating

Update (2020-11-22):

I just switched this blog to use Amazon’s Cloudfront CDN to speed up things (the old server was in Mt Prospect, Illinois near Chicago O’Hare airport, and particularly painful to access from London due to the latency). Unfortunately that also means the perfect A+ is gone, as AWS has more lenient compatibility settings for its SSL gateways.

On the bugginess of El Capitan

I never updated my home Mac Pro to El Capitan. To paraphrase Borges, each successive Apple OS release since Snow Leopard makes you long for the previous one. Unfortunately I have no choice but to run the latest OS X release on my work Macs as that is usually required to run the latest Xcode, itself required for the App Store.

I did not realize how bad El Capitan was until I upgraded my work iMac (27-inch 5K model) to Sierra last week. Previously, I would experience a mean time between crashes of around 3 days. I thought it was flaky hardware (the problems started from when I unboxed the computer), but couldn’t find the time to take it to the Genius Bar. I had also experienced the same problem with my old home 2009 Nehalem Mac Pro, which I had taken to the office, in fact that’s why I bought the iMac in the first place (and the first one I ordered had to go back because of defective pins in the RAM expansion slots). The Mac Pro had previously been rock-steady at home.

Since I upgraded to Sierra, I haven’t had a single crash. The only possible conclusion is that El Capitan bugs were to blame. The only thing unusual about this iMac is I upgraded the RAM from OWC, but the memory passes testing using Micromat’s TechTool.

I am not one to look at the Steve Jobs era with rosy-tinted glasses, OS X has never had the same level of stability as Solaris or even Linux, but Apple’s hardware and software quality has really gone to the dogs of late, something Lloyd Chambers dubs Apple Core Rot.

I am now starting to hedge my bets and am testing Ubuntu for my laptop computing needs, first by repurposing my 2008-vintage first-generation MacBook Air that is no longer supported by OS X anyway (works, but painfully slow) and soon with a shiny new HP Spectre on order.

Avoiding counterfeit goods on Amazon: mission impossible?

I mentioned previously that I seldom shop for electronics on Amazon.com any more, preferring B&H Photo whenever possible. I now have another reason: avoiding counterfeit goods.

My company boardroom is in an electromagnetic war zone—dozens of competing WiFi access points combined with electronic interference from the US-101 highway just outside make WiFi reception tenuous at best, and unusable more often than not. To work around this, we set up a wired Ethernet switch, and since most of our staff use MacBook Airs, Apple USB Ethernet adapters purchased from Amazon. When I side-graded from my 15″ Retina MacBook Pro to a much more portable 12″ Retina MacBook, I wasn’t able to connect using the dongle, and the name of the device was interspersed with Chinese characters. At first I thought it was an issue with my Satechi USB-C hub, but I experienced the same problems via a genuine Apple USB-C multiport adapter as well.

Eventually I figured out the Ethernet dongles were counterfeit. The packaging, while very similar to Apple’s, was just a tiny bit off, like amateurish margins between the Apple logo and the edges of the card. On the dongles themselves, the side regulatory disclosures sticker was inset, not flush with the body of the adapter.

Counterfeiting is a major problem. By some accounts, one third of all Sandisk memory cards worldwide are counterfeits. In some cases like chargers or batteries, your equipment could be at risk, or even your very life. The counterfeit adapters we purchased from Amazon did not come from Amazon themselves but from a third-party merchant participating in the Amazon marketplace. To Amazon’s credit, we returned them for a prompt, no questions asked refund even though we bought them over six months ago, but it is hard to believe Amazon is unaware of the problem rather than willfully turning a blind eye to it.

My first reaction was to tell our Office Manager to make sure to buy only from Amazon rather than third-party merchants (pro tip: including “amazon” in your Amazon search terms will do that in most cases). Unfortunately, that may not be enough. Amazon has a “fulfilled by Amazon” program for merchants where you ship your goods to them, and they handle warehousing and fulfillment. These “fulfilled by Amazon” items are also more attractive to Prime members. One option Amazon offers is Stickerless, commingled inventory where the items you send are put into a common bin. Amazon still has the ability to trace the provenance of the item through its inventory management, but for purposes of order fulfillment they will be handled just like Amazon’s own stock. Some categories like groceries and beauty products are excluded, but electronics are not.

The implications are huge: even if the vendor is Amazon itself, you cannot be sure that the item is not counterfeit. All the more reason to buy only from trustworthy, single-vendor sites like B&H, even if shipping is a bit slower.