Fazal Majid's low-intensity blog

Sporadic pontification

Automating Epson SSL/TLS certificate renewal

Network-capable Epson printers like my new ET-16600 have a web-based user interface that supports HTTPS. You can even upload publicly recognized certificates from Let’s Encrypt et al, unfortunately the only options they offer is a Windows management app (blech) or a manual form.

When you have to upload this every month (that’s when I automatically renew my Let’s Encrypt certificates), this gets old really fast, and strange errors happen if you forget to do so and end up with an expired certificate.

I wrote a quick Python script to automate this (and yes, I am aware of the XKCDs on the subject of runaway automation):

#!/usr/bin/env python3
import requests, html5lib, io

URL = 'https://myepson.example.com/'
USERNAME = 'majid'
PASSWORD = 'your-admin-UI-password-here'
KEYFILE = '/home/majid/web/acme-tiny/epson.key'
CERTFILE = '/home/majid/web/acme-tiny/epson.crt'

########################################################################
# step 1, authenticate
jar = requests.cookies.RequestsCookieJar()
set_url = URL + 'PRESENTATION/ADVANCED/PASSWORD/SET'
r = requests.post(set_url, cookies=jar,
                  data={
                    'INPUTT_USERNAME': USERNAME,
                    'access': 'https',
                    'INPUTT_PASSWORD': PASSWORD,
                    'INPUTT_ACCSESSMETHOD': 0,
                    'INPUTT_DUMMY': ''
                  })
assert r.status_code == 200
jar = r.cookies

########################################################################
# step 2, get the cert update form iframe and its token
form_url = URL + 'PRESENTATION/ADVANCED/NWS_CERT_SSLTLS/CA_IMPORT'
r = requests.get(form_url, cookies=jar)
tree = html5lib.parse(r.text, namespaceHTMLElements=False)
data = dict([(f.attrib['name'], f.attrib['value']) for f in
             tree.findall('.//input')])
assert 'INPUTT_SETUPTOKEN' in data

# step 3, upload key and certs
data['format'] = 'pem_der'
del data['cert0']
del data['cert1']
del data['cert2']
del data['key']

upload_url = URL + 'PRESENTATIONEX/CERT/IMPORT_CHAIN'

########################################################################
# Epson doesn't seem to like bundled certificates,
# so split it into its componens
f = open(CERTFILE, 'r')
full = f.readlines()
f.close()
certno = 0
certs = dict()
for line in full:
  if not line.strip(): continue
  certs[certno] = certs.get(certno, '') + line
  if 'END CERTIFICATE' in line:
    certno = certno + 1
files = {
  'key': open(KEYFILE, 'rb'),
}
for certno in certs:
  assert certno < 3
  files[f'cert{certno}'] = io.BytesIO(certs[certno].encode('utf-8'))

########################################################################
# step 3, submit the new cert
r = requests.post(upload_url, cookies=jar,
                  files=files,
                  data=data)

########################################################################
# step 4, verify the printer accepted the cert and is shutting down
if not 'Shutting down' in r.text:
  print(r.text)
assert 'Shutting down' in r.text
print('Epson certificate successfully uploaded to printer.')

Update (2020-12-29):

If you are having problems with the Scan to Email feature, with the singularly unhelpful message “Check your network or WiFi connection”, it may be the Epson does not recognize the new Let’s Encrypt R3 CA certificate. You can address this by importing it in the Web UI, under the “Network Security” tab, then “CA Certificate” menu item on the left. The errors I was seeing in my postfix logs were:

Dec 29 13:30:20 zulfiqar mail.info postfix/smtpd[13361]: connect from epson.majid.org[10.0.4.33]
Dec 29 13:30:20 zulfiqar mail.info postfix/smtpd[13361]: SSL_accept error from epson.majid.org[10.0.4.33]: -1
Dec 29 13:30:20 zulfiqar mail.warn postfix/smtpd[13361]: warning: TLS library problem: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca:ssl/record/rec_layer_s3.c:1543:SSL alert number 48:
Dec 29 13:30:20 zulfiqar mail.info postfix/smtpd[13361]: lost connection after STARTTLS from epson.majid.org[10.0.4.33]
Dec 29 13:30:20 zulfiqar mail.info postfix/smtpd[13361]: disconnect from epson.majid.org[10.0.4.33] ehlo=1 starttls=0/1 commands=1/2

Update (2021-08-01):

The script was broken due to changes in Let’s Encrypt’s trust path. Seemingly Epson’s software doesn’t like certificates incorporating 3 PEM files and shows the singularly unhelpful error “Invalid File”. I modified the script to split the certificate into its component parts. You may also need to upload the root certificates via the “CA Certificate” link above. I added these and also updated the built-in root certificates to version 02.03 and it seems to work:

  • lets-encrypt-r3-cross-signed.pem 40:01:75:04:83:14:a4:c8:21:8c:84:a9:0c:16:cd:df
  • isrgrootx1.pem 82:10:cf:b0:d2:40:e3:59:44:63:e0:bb:63:82:8b:00
  • lets-encrypt-r3.pem 91:2b:08:4a:cf:0c:18:a7:53:f6:d6:2e:25:a7:5f:5a

They are available from the Let’s Encrypt certificates page.

Apple iCalendar's buggy SNI

TL:DR If you use Apple’s calendar client software, do not run the server on an IP and port shared with any other SSL/TLS services.

I run my own CalDAV calendar server for my family and for myself. For a very long time I used DAViCal, but it’s always been a slight annoyance to set up on Apple devices because they don’t like DAViCal’s https://example.com/davical/caldav.php/majid URLs. What’s more, recent versions of iCalendar would pop up password prompts at random, and after re-entering the password a couple of times (once is not enough), would finally go on and work. The various devices would also all too often get out of sync, sometimes with the inscrutable error:

Server responded with “500” to operation CalDAVAccountRefreshQueueableOperation

requiring deleting the calendar account and recreating it by hand.

I tried replacing DAViCal with Radicale today, with the same flaky user experience, and I finally figured out why: Apple uses at least a couple of daemons to manage calendar and sync, including dataaccessd, accountsd and remindd (also CalendarAgent depending on your OS version). It seems some or all of them do not implement Server Name Indication (SNI) consistently. SNI is the mechanism by which a TLS client indicates what server it is trying to connect to during the TLS handshake, so multiple servers can share the same IP address and port, and is an absolutely vital part of the modern web. For example many servers use Amazon Web Services' Elastic Load Balancer or CloudFront services, which are used by multiple clients, if Amazon had to dedicate a separate IP address for each, it would break their business model1.

Sometimes, those daemons will not use SNI, which means they will get your default server. In my case, it’s password-protected with a different password than the CalDAV one, which is what triggers the “enter password” dialog. At other times, they will call your CalDAV server with dubious URLs like /.well-known/caldav, /principals/, /dav/principals/, /caldav/v2 and if your server has a different HTTP password for that and sends back a HTTP 401 status code instead of a 404 Not Found, well, that will also trigger a reauthentication prompt.

Big Sur running on my M1 MacBook Air seems to be more consistent about using SNI, but will still poke around on those URLs, triggering the reauthentication prompts.

In other words, the only way to get and Apple-compatible calendar server running reliably is to dedicate an IP and port to it that is not shared with anything else. I only have one IP address at home where the server runs, and I run other vital services behind HTTPS, so I can’t dedicate 443 to a CalDAV server. Fortunately, the configuration will accept the syntax example.org:8443 to use a non-standard port (make sure you use the Advanced option, not Automatic), but this is incredibly sloppy of Apple.


  1. Amazon does in fact have a Legacy Clients Support option, but they charge a $600/month fee for that, and if you need more than two, they will demand written justification before approving your request. ↩︎

Edgewalker, a DIY VPN server

TL:DR Don’t trust VPN services, roll your own with this easy script.

Rationale

There are many reasons to use a Virtual Private Network. Perhaps you are on an unsecured WiFi network. Perhaps you don’t want your Internet Service Provider to snoop on your browsing history using Deep Packet Inspection and compile a marketing dossier on your. Perhaps like my daughter you want to access video content on Netflix that is not available in your country. Perhaps you want to bypass the nanny state content filters the British government mandates.

Most VPN services are untrustworthy. You depend on the VPN provider’s assurances to protect your privacy, which completely defeats the purpose of a VPN. The only way you can be sure is to run your own, but baroque network protocols engendering complex software makes it difficult to do so even for the technically savvy.

Streisand was one of the first efforts to automate the process, using cloud virtual servers as the hosts operating the VPN. Trail of Bits implemented Algo to simplify it and remove some questionable choices Streisand made (although, to be fair, the Streisand project seems to have jettisoned many of them and converged on WireGuard).

Edgewalker is similar, but awesomer:

  • It is based on OpenBSD, widely considered the most secure general-purpose OS, rather than Linux.
  • Like Algo, it implements IPsec/IKEv2/MOBIKE rather than OpenVPN (read the Algo announcement for the reasons why).
    • IPsec/IKEv2 works out of the box on iOS, iPadOS and macOS.
    • In theory on Windows as well, although I have no idea how to make it work or simplify setup, any help is welcome.
  • It also implements WireGuard (recommended for Linux and Android, along with travel VPN-capable routers like the GL.iNet Mango)
  • It uses QR codes to simplify installation as much as possible on the client devices.
  • It uses Let’s Encrypt so your IPsec certificates just work (WireGuard does not rely on PKI)
  • It uses its own Unbound DNS server with DNSSEC validation support, for better privacy
  • It has no dependencies on Ansible, Python or anything else exotic you need to add on your own machine, other than a SSH client.
  • It is just a shell script with little bits of Python thrown in like Acme-Tiny, and easily auditable.

While you can run the script again as your Let’s Encrypt certificates expire (although it generates new credentials each time), I recommend simply destroying the VM and creating a new one. Of course, if you are running on physical hardware, you will want to rerun the script. If using WireGuard only, you don’t need to rerun the script as WireGuard keys do not expire and there are no certificates.

Prerequisites

You need:

  • A Let’s Encrypt account and key (I’m working on setting this up automatically for you, in the meantime you can use Step 1 on this page to do that for you).
  • An OpenBSD machine reachable from the Internet (it can be a physical machine you own, or a cloud VM like Vultr).
  • The ability to add a DNS record for the machine’s IP address (IPv4 only for now).
  • The 80x25 OpenBSD console does not support UTF-8 and cannot display the QR code in a single screen. Use a different terminal, or enter the profile URL by hand.

If you have a firewall in front of the OpenBSD machine, it needs to allow the following inbound traffic (possibly using static port mappings if you use NAT):

  • SSH (TCP port 22) so you can actually log in to your machine.
  • HTTP (TCP port 80) and HTTPS (TCP port 443) to allow Let’s Encrypt certificate issual and allow you to get the Apple-format Profiles that will ease setup on your iDevice.
  • UDP ports 500 (IKE), 1701 (IPsec) and 4500 (IPsec NAT traversal).
  • Optionally IPsec protocols ESP (IP protocol number 50, hex 0x32)) and AH (decimal 51 hex 0x33) and ESP for maximum efficiency, although many firewalls won’t support this.
  • UDP port 51820 (WireGuard).

Instructions

  • Clone the Github repository into one of your own, or copy the file edgewalker.sh somewhere you can download it without it being tampered with in transit, in practice that means HTTPS.
  • Edit the first lines in the script edggewalker.sh (X509 and USERNAME). Not strictly necessary, but make it your own.
  • Log in as root on your OpenBSD machine, then:
    pkg_add wget
    wget -c https://raw.githubusercontent.com/YOUR_GITHUB_ACCOUNT_HERE/edgewalker/main/edgewalker.sh
    sh -e edgewalker.sh
    
  • The script will ask you for:
    • The DNS name of your OpenBSD machine.
    • To copy-paste your Let’s Encrypt account key in PEM format.
  • It will then obtain Let’s Encrypt certificates, generate a QR code that you can use to download the profile on your iDevice to set up the VPN.

Credits

  • The OpenBSD team, for making their wonderful security-focused OS.
  • Reyk Flöter for making OpenIKEd, a breath of fresh air in the unnecessarily convoluted world of VPN software.
  • Jason A. Donenfeld for inventing WireGuard.
  • Let’s Encrypt, for making certificates cheap and easy.
  • Daniel Roesler for the fantastic Acme-Tiny.

Demo

I created a fresh OpenBSD 6.8 VM vpn42.majid.org on Vultr, and here is what the experience looks like:

Here is how to install the VPN on an iPhone:

Here is how to create a suitable VM on Vultr:

Canon Powershot Zoom review

Exporting secrets from the Lockdown 2FA app

The Lockdown app mentioned in this article was last updated in 2015, and if you don't already use it, I would not recommend your adopting it.

I am (very) slowly migrating away from the Mac to Ubuntu Linux as my main desktop operating system. The reasons why Apple has lost my confidence are:

  • The execrable software quality of recent releases like Catalina (I plan on sticking with Mojave until I have migrated, however long that takes).
  • Apple’s increasing locking down of macOS in ways antithetical to software freedom, e.g. SIP or the notarization requirements in Catalina with the denial of service implications
  • The fact they no longer even pretend not to price-gouge on the Mac Pro. My days of buying their professional workstations every 5 years have come to an end after 15 years (PowerMac G5, Nehalem Mac Pro, 2013 Mac Pro)
  • As the iPhone market is saturating, in their eagerness to come up with a replacement growth engine in “services”, they are pushing app developers towards the despicable and unacceptable subscription licensing model
  • The butterfly keyboard fiasco exemplifies the contempt in which the company seems to hold its most loyal customers

On the plus side, ThinkPads have decent keyboards, unlike all Apple laptops since at least 2008, the LG Gram 17 is both lightweight, powerful and its huge screen is a boon to my tired eyes, and I am favorably impressed with the deep level of hardware integration offered by Ubuntu (e.g. displaying the logo and boot status in the UEFI stages of boot), even if I am not enamored of the software bloat or systemd.

One of the tasks in my migration checklist is to find a replacement for my TOTP two-factor authentication solution, which is currently the Lockdown app on iOS, iPadOS and MacOS (based on this recommendation, not to be confused with the Lockdown firewall/VPN app). I don’t trust Authy, they have a record of security failures introduced by their attempts to extend standard TOTP with their proprietary garbage, but I digress…

Thus I need to export Lockdown secrets. The iOS app can print or email a PDF with QR codes as a backup, but that’s not a very usable format for migration.

As I had to add a new TOTP secret to the app recently, that was the impetus to do this as a weekend project. I implemented a small utility called ldexport in Go to decode Lockdown for Mac’s internal file into either JSON or HTML format. Here are some simulated samples:

[
    {
        "Service": "Amazon",
        "Login": "amazon@example.com",
        "Created": "2015-11-18T19:53:34.969532012Z",
        "Modified": "2015-11-18T19:53:34.969532012Z",
        "URL": "otpauth://totp/Amazon%3Aamazon%40example.com?secret=M7IoBWqA2WuzYG27ju82XTWsflPEha3xBafMQ3i9CgwKgp6RdBGh\u0026issuer=Amazon",
        "Favorite": true,
        "Archived": false
    },
    {
        "Service": "PayPal",
        "Login": "ebay@example.com",
        "Created": "2019-11-25T08:46:57.253684043Z",
        "Modified": "2019-11-25T08:46:57.253684043Z",
        "URL": "otpauth://totp/PayPal:ebay@example.com?secret=3gB0VWJFkaYcVIiD\u0026issuer=PayPal",
        "Favorite": false,
        "Archived": false
    },
    {
        "Service": "Reddit",
        "Login": "johndoe",
        "Created": "2020-08-07T19:58:37.930042982+01:00",
        "Modified": "2020-08-07T19:58:37.930042982+01:00",
        "URL": "otpauth://totp/Reddit:johndoe?secret=nDTxDMI6bEgVpHWCViZjDFhXKH1bysRa\u0026issuer=Reddit",
        "Favorite": true,
        "Archived": false
    },
    {
        "Service": "GitHub",
        "Login": "",
        "Created": "2016-05-04T19:04:12.495128989+01:00",
        "Modified": "2017-04-04T06:33:10.641680002+01:00",
        "URL": "otpauth://totp/github.com/johndoe?issuer=GitHub\u0026secret=bXh5qmeTMzcatKKz",
        "Favorite": false,
        "Archived": false
    },
    {
        "Service": "Google",
        "Login": "johndoe@gmail.com",
        "Created": "2015-11-13T05:06:07.103500008Z",
        "Modified": "2015-11-13T05:06:07.103500008Z",
        "URL": "otpauth://totp/Google%3Ajohndoe%40gmail.com?secret=o5MvqdWDt7ZEHHSTuH6rCAUr4M6ozGQD\u0026issuer=Google",
        "Favorite": false,
        "Archived": false
    }
]