Fazal Majid's low-intensity blog

Sporadic pontification

HSTS: surprisingly rare

HTTP Strict Transport Security (HSTS) is a critical security feature that allows a site to say “always use the secure HTTPS version, not the insecure unencrypted one”. There is a chicken-and-egg effect where the first time you access a website, you have no way to know if your site has HSTS turned on or not without accessing it, so browsers distribute a “HSTS Preload” list of domains for which it is turned on even if you have never accessed it before, as explained by Adam Langley of the Google Security Team. On Chromium based browsers you can check by accessing chrome://net-internals/#hsts. Yours truly is on the list, which means that almost every single device on the planet has a file with my name in it, to my never-ceasing amusement.

Someone asserted that most e-commerce and financial sites are registered with HSTS Preload. I have a pretty jaundiced view of banks’ security, the fact most of them consider sending 6-digit codes by SMS a valid form of two-factor authentication leads me to believe they mostly engage in security theater. So I used the official Google Chrome HSTS Preload portal to check.

I was shocked to find out that in fact not only is HSTS Preload very rare, but even HSTS itself is hardly present. None of the sites I checked use either:

Not even Amazon.com has it, despite being a company that operates a Certificate Authority.

The only explanation I can think of is that this is a deliberate product decision to make life easier on those annoying free WiFi with captive portals, at the expense of security.

Captive portals are those WiFi networks that don’t support IEEE 802.11u Hotspot 2.0, which means that instead of showing you a popup when you connect to WiFi asking you to agree to the terms of service, sign in to a paid WiFi service or whatever, it will instead hijack the first non-TLS HTTP request and show you the captive portal page instead (pro tip: use neverssl.com as the first page you access on those portals). If you were to only access https://amazon.com/, you would hang forever, whereas with http://amazon.com/ you would first get the captive portal page, then on reload the actual Amazon page.

The flip side is that anyone can set up a WiFi pineapple and SSLstrip in a Starbucks to impersonate their free WiFi, hijack your connection by issuing a deauthentication frame to force you to disconnect from Starbucks’ WiFi and connect instead to your fake Starbucks WiFi, and then the attacker can use the SSL stripping described by Adam Langley to steal your Amazon password, even if you have two-factor authentication enabled. Given how easy Amazon has made it to impersonate them, I am surprised this kind of scam is not more prevalent.

My Broadband Setup

TL:DR Setting up secure and resilient Internet access in a country with sub-par infrastructure

I moved to the UK, a country that was a leader in Europe for PC adoption and early telecoms deregulation, but has since become one of the worst for the quality of its broadband through misguided laisser-faire policies. The only fixed broadband option available in my apartment is BT OpenReach’s pathetic VDSL service1 (resold by Vodafone), which advertises 72 Mbps but I am lucky to get 40 Mbps down and 10 Mbps up.

There are several problems with this state of affairs:

  • The network is very unreliable. I’ve had outages lasting 8 hours. It is so bad I wrote my own tool to track ping times and downtime.
  • The consumer ISPs in the UK are anything but network-neutral, due to government regulations mandating Orwellian nanny-filters on the connection2. At one point, I was unable to reach the Stack Overflow for over 2 days. It turns out for some unfathomable reason Vodafone decided to use Stack Overflow as the test site when they developed the government-mandated nanny-filter, and somehow that was deployed to production as per this highly instructive email thread.
  • The IP address is dynamic. While Vodafone does not change it too often and it can be worked around using Dynamic DNS, on cellular carriers the use of Carrier-Grade NAT (CGNAT) is rife, and it makes those connections highly unsuitable for:
    • self-hosting mail servers, calendar or other services
    • working from home where I need to have long-lived SSH connections doing critical work.

Recently I found out my mobile operator, Three, offers 5G fixed broadband service. I was skeptical, their 4G service in my NIMBY-infested area3 is abysmal, I hardly ever have any signal at all on Hampstead High Street, but it turns out their 5G service is excellent, offering 500 Mbps down and 30 Mbps up, with decent ping times, probably because they managed to buy a 100Mhz contiguous allocation of 5G spectrum. Unfortunately, the service is not officially offered in my post code, so I decided to roll my own using an unlimited SIM card and an unlocked Huawei CPE Pro 2 5G router.

I have been experimenting with VPNs of late, leading to my edgewalker self-hosted VPN server, and building a VLAN on my network that thinks it is in the US using a VPN provider that shall remain nameless because it still has not been blocked by Netflix. This allows my daughter to watch her favorite US shows that are not available in the UK because of the despicable geofencing of the content cartels, who want to gouge you depending on where you live (except in this case they are not even offering the gouging, just no content).

The natural next step is to make the entire network be connected to the Internet via a WireGuard VPN. Because WireGuard, was designed for mobile connections like IPsec/IKEv2/MOBIKE, it easily adapts to shifting IP addresses (as long as one end stays put). This means it can deal with CGNAT and also fail over from 5G to DSL and back without breaking a sweat or even dropping a session.

Unfortunately there are side-effects to using a self-service VPN hosted by a cloud provider:

  • Netflix, Amazon and the BBC will refuse to serve video to you. I had to work around it by creating a special VLAN for VPN-averse devices (the LG Smart TV, AppleTV 4K and any other streamers in my household). This VLAN is bridged to the Huawei in a way that stops the offensive STP packets, so it is as if they were plugged in directly into the Huawei. This is not a solution for when we want to watch video from out iDevices, however.
  • The VPN encapsulation reduces the maximum data size (MTU) from the standard 1500 bytes of Ethernet to 1380 or so. Some sites have broken Path MTU Discovery (I found out the hard way DuckDuckGo is one of them), which means by blocking ICMP packets the server does not realize their large packets are not getting through, keeps retrying in vain until the browser times out in disgust. Setting the OpenBSD PF scrub (no-df) option took care of that.
  • Then there is the bizarre phenomenon by which Google thinks my IP is in the United Arab Emirates. I do not know how, IP2Location thinks it is in the Netherlands, and MaxMind that it is in the UK (as it is). I tried again with some other Vultr servers and kept being located to the UAE or Saudi Arabia. My best guess is that Google builds its own IP geolocation database using GPS data from Android phones, and that some brave souls in the UAE or Saudi Arabia used a VPN service running on Vultr servers, and that caused the Vultr IPs to be associated with the those countries. The only way I found to resolve that was to keep creating virtual servers and additional IPs until I found a pair that did not locate to the UAE or Saudi Arabia. Now Google thinks I am in the US rather than the UK, I can live with that.
  • Some services like Wikipedia will also block the device from edits, as it triggers a false positive for an open proxy. I sent an email to them on Saturday night and they had fixed that by next morning, whereas Google makes strenuous efforts to ensure you cannot reach a human within their organization, ever, and there is seemingly no way to prevent the defective IP geolocalisation from screwing things up (they disabled the /ncr workaround they used to have a few years ago). Tells you everything you know about the importance of customer service for a monopoly.

This is what I implemented, by replacing the too-limiting Ubiquiti Security Gateway in my UniFi switched and wireless network with an OpenBSD router that establishes a WireGuard VPN to a modified edgewalker running in the cloud with Vultr.

The configuration is quite complex because I have the following VLANs:

  • The default VLAN (which is actually not even a VLAN, as Ubiquiti gear is not really Enterprise-class and does not default to VLANs). Because of the VPN compatibility issues, I am now using source IP based policy routing on the default VLAN and only servers go through the VPN.
  • VLAN 2 for my office work-from-home Mac, I just do not trust the various antivirus (and other software that are required for compliance) anywhere near my personal networks
  • VLAN 4 for the VPN-averse devices as mentioned above
  • VLAN 7 is directly on the ISP router
  • VLAN 666 for Internet of Things devices (at least those that can be operated without connecting to the rest of the LAN unlike printers)
  • VLAN 1776 for my geofencing-busting freedom VPN that thinks it is in the USA
  • Not a VLAN, but the Ethernet connection between my OpenBSD box and the Huawei router runs on a dedicated interface because in a bizarre effort to be “helpful” it sends a stream of Spanning Tree Protocol (STP) packets that basically cause my Ubiquiti switched network to melt down. OpenBSD can block them, but seemingly UniFi does not give you that control (so much for security, then). VLAN 4 is bridged to this.

OpenBSD has a concept of routing domains that allows you to virtualize your network stack into multiple routing tables, the way you can with VRF on a Cisco. This has proved invaluable, as has managing the configuration files in git to ensure I can always back out failed changes, and using Emacs’s TRAMP mode to edit files remotely.

It is mostly running, I have yet to move the Vodafone VDSL PPPoE circuit over from the decommissioned USG to the OpenBSD router and set up an IGP or some other routing protocol to fail over the default route to the Internet underlying WireGuard if one of the two connections fails. I am sure I will discover oddities as I go.

5G is extremely sensitive to positioning. Moving the Huawei just 20cm along the window makes the difference between 300Mbps down/10Mbps up/20ms ping and 500/30/12ms.

Not everything is perfect, of course. Ping times have risen slightly, and are more variable, as can be expected of a wireless network with layers of VPN processing latency added.

All DNS has to go through my own DNS servers, which are routed through the VPN so my ISPs cannot sniff those requests, or tamper with them. I thus get the benefits of encrypted DNS without having to trust the likes of Cloudflare or worse, Google. I do not (yet) block DoH or DoT like this gentleman does but I am planning to. My endgame is to add filtering similar to what the PiHole does, and interface the DNS server with the firewall to only allow IoT devices to connect to IP addresses that are the result of legitimate DNS lookups. I am also planning on recording DNS lookups using the dnstap interface for audit and parental control purposes.

Each VLAN has its own DHCP server instance. Known MAC addresses get a static IP in their DHCP lease, others get an address in and all of their network traffic is recorded for forensics purposes using pflog. The same also applies to IoT devices.

Ubiquiti devices are prohibited by the OpenBSD firewall from accessing the Internet by the firewall, because of their history of security breaches, and to prevent any interference from the cloud.

Update (2022-06-10):

Real fiber finally reached my apartment complex in October 2021. It’s operated by a small Manchester ISP called 4th Utility. They top at 400Mbps but it is symmetrical. Annoyingly, their routers are locked down and you need to ask customer support to make any changes. I was the first one connected, but sadly for them the median age here is probably 70 and they are having a hard time signing up new subscribers.

  1. It is a travesty that the Advertising Standards Authority has allowed ISPs to deceptively advertise their lousy copper DSL networks as “full fibre” on the basis they have fiber somewhere, and that this was not laughed out of court. ↩︎

  2. The UK is not quite as bad an enemy of the Internet as Australia, but only just. After all, this is a country without a Constitution, without a Bill of Rights or separation of Church and State, with a monarchy that is far from merely ceremonial, and where the ruling party campaigned on a manifesto of “we need to cut back on human rights”. ↩︎

  3. NIMBYs do not like cellular towers and even Uber drivers remark on how bad reception is in Hampstead. ↩︎

A pack of... backpacks

TL:DR I try way too many backpacks so you don’t have to

I have many bags. So many I no longer keep an inventory in a spreadsheet but use a relational database to track. For a very long time, I preferred messenger bags but at age 36 I started developing muscle spasms in the right shoulder. After a few years of off and on physical therapy, I figured out it was the asymmetric load from the shoulder bag that was causing it (even though the load was on the left shoulder). This left me no option but to switch to backpacks exclusively, despite their less than ideal looks.

There are many reviews on the web, including on YouTube, but most are influencer shills who will not disclose the flaws of the bags, or simply don’t use the evaluation copies long enough to find out. Some sites like Carryology have inherent conflicts of interest because they share ownership with a manufacturer (Bellroy), and surprise surprise, those dominate the Best Of rankings, go figure.

You can get more honest feedback on the many Reddit bag-related forums (r/backpacks, r/ManyBaggers, r/onebag) or on blogs, but it does require wading through post after post.

EDC Loadout

Here are my reviews on backpacks I have actually owned and used. But before we start, you need to know what I carry in them to assess whether my needs are congruent to yours:

  • 13″ MacBook Air
  • Sometimes a 17″ LG Gram 17 instead (running Linux, of course)
  • 12.9″ iPad Pro 2020
  • A Tech Dopp Kit
  • A full-size mirrorless camera with mid-size lens (Nikon Z7, Fuji X-T4 or Leica M11)
  • Apple AirPod Max or Sony WH-1000XM3 noise-cancelling headphones in their case
  • London Undercover folding umbrella (I live in rainy London now…)
  • Sometimes a Bluetooth mechanical keyboard (Keychron K7 in its fitted leather case)
  • Zeiss Victory Pocket 8×25 binoculars, or if this is a serious birdwatching trip, Swarovski NL Pure 8×42
  • a water bottle
  • a first-aid kit

Features I look for

  • Comfort
    • The quality of the straps (specially relevant for women and their distinct upper-body anatomy)
      • How well are they contoured and padded?
      • Whether they have a sternum strap or not
    • The material on the back, is it breathable, specially in warmer climes?
    • Does the weight rest in the right position on your back?
  • Quality of materials
    • Water-resistance (taped seams, AquaGuard zippers)
    • Abrasion resistance
    • Is the material too rough and will it scuff your clothes, e.g. certain grades of Cordura?
    • Is it pleasant to the hand and looks good?
    • Better technical materials like Dimension Polyant X-PAC or DSM Dyneema combine strength with light weight
    • Zippers: YKK is a good choice, or higher-end brands like RiRi. No-name zippers are a warning sign
    • Quality of stitching, e.g. bar-tacking in stress points
    • Quality of hardware, e.g. metal instead of plastic, or premium hardware like Fidlock or Austri-Alpin buckles
    • Velcro is usually a bad sign, it is noisy, collects lint
  • Capacity
    • Bag makers are surprisingly bad at estimating the capacity of their bags, even though there is an official ASTM standard for this
    • Resist the temptation to overpack
  • Organization
    • Ease of access and packing using a full-clamshell design
    • Beware of excessively organized bags
      • When you don’t need all of the organization, it still adds weight and reduces the usable space in compartments.
      • Several smaller compartments are less versatile than a single larger, less organized but more flexible compartment that can take odd-sized items like a camera, full-sized headphones or shopping
      • I have never understood the point of cell phone pockets in a bag. By the time you take the bag off, open it and extract your phone, surely the call has gone to voice mail?
    • Laptop compartment
      • Is the laptop suspended? If not, and you put the backpack down on the floor abruptly, the laptop will hit the hard floor and sustain damage
      • Are there metallic zipper teeth that could scratch your laptop?
      • Are the zippers waterproof, e.g. YKK AquaGuard? If not, water could get in and damage your laptop

DSPTCH Ridgepack Dyneema ★★★★★

A very light minimalist backpack with a distinctive silhouette. The large main compartment lets you organize as you see fit and is very versatile, with a laptop sleeve if you need one. A full clamshell YKK Aquaguard zipper ensures water-resistance, but it is also harder to open than conventional zippers (pro tip: fold back the rain flap that shields them to make opening it easier). One corner cut that should not have in a bag this price: the plastic hardware and using a cheap Duraflex buckle in the sternum strap instead of a Fidlock. One strange touch is the detachable clips at the top of the shoulder straps. They serve no discernible purposes and make them susceptible to twisting, and probably reduce durability. Made in the USA.

DSPTCH RND Daypack Dyneema ★★★

A larger work-oriented backpack with a separate laptop compartment and large capacity. It has the same disappointing cheap hardware as in the Ridgepack and the same shoulder strap clips. The bag does not have a full clamshell opening, that makes it harder to pack or to access contents

Able Carry Max ★★★★

A large bag for when that is called for, even if I doubt it actually has 30L capacity, seems more like the 26L GoRuck GR1. It is made of quality materials (X-PAC, but with a more abrasion-resistant Cordura bottom), and available in colors other than boring black (I have it in green, even if is more of a dark khaki).

The water bottle pocket is excellent, large enough to hold a champagne bottle, or more to the point, a large folding umbrella.

Black Ember Shadow 26L ★★★★

This bag has the same apparent capacity as my 19L Brown Buffalo, I would say it is a 20L bag, certainly not 26L. Less structured than the Citadel, with more usable space. The material is a fine-denier ballistic of some sort, not a slick coated tarp-like fabric like on the Citadel. The built-in nonremovable tech organizer, somewhat reminiscent of the Peak Design Tech Pouch in its alternating pocket design, is a polarizing feature. It does obstruct the opening of the bag a bit, and the retaining strap could be secured to the flap better.

Black Ember Citadel Minimal V2 ★★★

A handsome, very organized bag, perhaps too much so. Unlike the Aer Tech Pack 2, it actually has a usable main compartment (as long as your laptop pocket is not too stuffed), and the quality of materials is better. Bonus points for the full-clamshell design and sternum strap. Very water-resistant (IPX7 rated, in fact). I still prefer the less structured Shadow.

Rofmia × Cathedral Shift Daypack V2 ★★★★★

This is a limited edition version of their Dyneema Daypack V2 in Dyneema Leather. Amazing material, but twice the price and less practical as it is heavier and less water-resistant, but the things we do for bragging rights…

Dyneema Leather has the crinkled Tyvek-like look of Dyneema. It is too thin to have the luxurious hand feel of leather, but it is certainly a kind of leather.

The bag is incredibly light, has interesting touches like a triple zipper and a collapsible internal water pocket. Sternum strap with fidlock, as could be expected for the price. It is by far my favorite EDC bag when it is not raining, and I got the regular Dyneema one for when it is.

Baron Fig Venture Slimline Backpack

A very slim backpack, meant to hold a laptop or notepad and not much else. Very basic straps (canvas webbing, no padding or sternum strap, and simplistic adjustment buckles, albeit metal). It’s made of canvas so I would expect zero water-resistance. I suspect if you are tall enough you can actually wear it concealed under your jacket or raincoat like the old Betabrand Under-the-Jack Pack.

Gitzo Century Traveler Backpack ★★★

A very interesting photo backpack, a much better execution of the Peak Design EDC backpack concept in my opinion. Has some smart touches like a tripod holder designed for the Gitzo Traveler mini-tripod (hence the name), or a stash for your lens cap.

The camera section has a removable insert whose sides can be unzipped for quick access from the sides of the backpack, a better design than the Peak Design. Unfortunately it has very limited space for stuff other than the camera and laptop, which limits its usefulness as a travel or EDC bag.

Mission Workshop Spar harness VX ★★★★

Very small backpack I bought on a hot summer day where wearing my usual jacket was not an option. Can barely hold a 12″ MacBook in its laptop sleeve, a 13″ MacBook Air or 12.9″ iPad Pro is out of the question. Surprisingly comfortable straps.

Bedouin Foundry Pequod ★★★

Top-quality materials as befits the price, leather, Dyneema and Austri-Alpin Cobra paragliding buckles. There is no way this is a 30L bag, 20L at most if that. Interesting tapering shape towards the bottom. Not incredibly practical but a looker.

UCON Acrobatics Alan bag, Olive ★★

My only roll-top backpack. Made of green neoprene. Tall but slim, moderate capacity, very water-resistant, but limited organization inside. Ultimately I hardly ever use it because the roll-top design, combined with a narrow and very tall bag, makes it hard to pack.

Tumi Mission Bryant leather backpack ★★★★

This was my daily work bag for a long time. It was made by Tumi before their acquisition by Samsonite after which quality has reportedly gone downhill. Bought on sale from Vente-Privée.com. Very good quality, large capacity, but currently in storage since I moved to the UK.

Knomo Albion, brown & black ★★★★

I have both the black and brown versions of this handsome full-grain leather bag from British brand Knomo, well known for its elegant women’s laptop bags, but that also has a line for men. The design is simple with fairly limited organization, but it has ample capacity and looks good, and the price is an outright steal for the quality (I paid $100 for my first on Massdrop and £134 for the second from their Covent Garden shop). Sadly it is discontinued, but some new-old-stock is still available online.

Capra Leather Tamarao Backpack, Hunter Green ★★★

A very large but very slim leather backpack made by Colombian artisans. I got the large one in hunter green (you can never be too rich or too green is my motto), it is really more of a dark olive green, and reasonably close to the product photos on my calibrated monitor.

The bag is much sleeker than I expected, about 10cm thin. Because it is the large size, the laptop pocket fits my LG Gram 17 perfectly, admittedly it is fairly small for a 17″ laptop. I am 1m81/6′, and I wouldn’t recommend the large size for someone shorter.

The leather quality is very good, I haven’t had the time to verify its water resistance. The visible stitching looks saddle-stitched to my untrained eyes. I opted for the baggage passthrough loop. It is made of black suede like the back lining of the bag, I am not sure it is that worthwhile an option.

The straps are straight and padded with suede, very basic and not contoured to fit your body shape. I think they were designed to look good when you carry the back by the hand strap.

The interior lining is a black linen material, not the medium gray shown on their website. On the plus side, that means stains won’t show, but it also means stuff is harder to find inside, although I am not sure how much that matters in a relatively small capacity bag like this.

Something to keep in mind: the bag doesn’t have an internal frame and the leather is soft, not stiff, so you would expect it to flop if not filled or at least with large items like a laptop or large sketchbook to keep its structure. I’m not sure what the purpose of the two zippers is on the back panel, they both open on the same small compartment. I suppose you could roll a jacket or sweater and slide it in there.

GoRuck GR1 Slick 26L ★★★

GoRuck bags have an enviable reputation for durability, but the tacticool (MOLLE and morale patch velcro) are a bit much for someone whose military service is 30 years in the past. The Slick version, available from Huckberry, drops those. It is a very large bag, with MOLLE inside you can attach admin pouches or organizers to, a much better approach than velcro in my opinion, even if it does take a while to attach. The laptop section is very well protected. That said it is very crude, from the sandpaper-like Cordura material, to the very plain zipper pulls (basically paracord tied at the ends with heat-shrink tubing) and other details.

The Brown Buffalo Conceal Backpack V3 ★★★★

I have the 19L version in X-PAC. The build quality is excellent, but the design is perfectible, and an already expensive bag is made more so by the fact no laptop sleeve is included. The front side-loading compartment is awkward to load a 13″ laptop into, and the velcro inside the main compartment (to attach organizers or the laptop sleeve) is the completely wrong approach as far as I am concerned. The two deep pockets are quite good, though, large enough to hold a big water bottle or full-sized keyboard. Unfortunately after the reboot of the company, the new versions have dropped the best features and kept the questionable ones.

Aer Flight Pack 2 ★★★

I have the X-PAC version (starting to sound like a refrain?). It is a good travel bag, the bright orange lining makes it easy to find things in, the design is not stiff and cramped like the Tech Pack 2. However the convertible design (so you can use it as a briefcase) is a bad idea, that means it cannot be a full clamshell and as neither fish nor fowl the design is compromised.

Chrome Hondo Welterweight Backpack ★★

Very boxy. I now use it primarily to stow some electronics test & measurement equipment (oscilloscope, power supply).

JanSport Mono Superbreak Mystic Pine ★★★

Cheap and cheerful (literally, a bright green) but has a surprisingly good warranty. Can’t be beat for value.

Arktype Design Dashpack Green waxed canvas LE ★★★

Very slim bag that discourages overpacking. The side-access compartment is on the small side and it is hard to insert a 13″ laptop without it catching. There is some MOLLE on the bottom, but not obnoxiously so. The rear compartment is designed to be used with the bag horizontal as you swing it, but that is not how I use a bag so it works at cross-purposes. Mine is the very short-lived green limited edition, a forest green in waxed canvas, quite good-looking. Sadly, I must dock points for the lack of a sternum strap. The compression straps on the side are completely useless and obstruct access to the water bottle pockets.

Aer Tech Pack 2 (no stars)

I had the Tech Pack 2, used it for a couple of weeks then sold it. It is very heavy, very stiff, and excessive organization means you end up with a lot of tiny inflexible compartments that won’t accommodate bulkier items like a DSLR or full-sized headphones. What’s worse, the tiny opening makes it very hard to access stuff, and unlike my Flight Pack X-PAC there is no bright orange lining to make things easy to find.

Peak Design Every Day Backpack

I have the Everyday V2. It’s not a good EDC bag at all and only a middling camera bag. The mesh fabric on the side flaps does not feel right. If you like the concept of mixed camera and EDC bag the Gitzo Century Traveler backpack is a much better option, with clever design touches like a tripod carrier and lens cap stash pocket.

Timbuk2 Blue Backpack ★★

A cheap and cheerful Timbuk2 backpack, don’t remember the model and it is probably discontinued anyway. Not much to say about it.

Moleskine Green Leather classic backpack ★★

A medium-sized backpack in an olive drab leather. The interior lining is a bit floppy and doesn’t seem all that durable. The bottom of the bag is molded EVA foam and looks tacky in comparison with the rest of the design.

Moleskine Green Leather Device Bag ★★

A small, very thin bag that is part vertical briefcase and part backpack. Nice green color, but little else to recommend it.

Porsche Design Backpack ★★★

One of the first bags I got. Small, trapezoidal design, quite elegant but the materials are fairly ordinary and the leather grab handle has cracked.

Automating Epson SSL/TLS certificate renewal

Network-capable Epson printers like my new ET-16600 have a web-based user interface that supports HTTPS. You can even upload publicly recognized certificates from Let’s Encrypt et al, unfortunately the only options they offer is a Windows management app (blech) or a manual form.

When you have to upload this every month (that’s when I automatically renew my Let’s Encrypt certificates), this gets old really fast, and strange errors happen if you forget to do so and end up with an expired certificate.

I wrote a quick Python script to automate this (and yes, I am aware of the XKCDs on the subject of runaway automation):

#!/usr/bin/env python3
import requests, html5lib, io

URL = 'https://myepson.example.com/'
USERNAME = 'majid'
PASSWORD = 'your-admin-UI-password-here'
KEYFILE = '/home/majid/web/acme-tiny/epson.key'
CERTFILE = '/home/majid/web/acme-tiny/epson.crt'

# step 1, authenticate
jar = requests.cookies.RequestsCookieJar()
r = requests.post(set_url, cookies=jar,
                    'INPUTT_USERNAME': USERNAME,
                    'access': 'https',
                    'INPUTT_PASSWORD': PASSWORD,
                    'INPUTT_ACCSESSMETHOD': 0,
                    'INPUTT_DUMMY': ''
assert r.status_code == 200
jar = r.cookies

# step 2, get the cert update form iframe and its token
r = requests.get(form_url, cookies=jar)
tree = html5lib.parse(r.text, namespaceHTMLElements=False)
data = dict([(f.attrib['name'], f.attrib['value']) for f in
assert 'INPUTT_SETUPTOKEN' in data

# step 3, upload key and certs
data['format'] = 'pem_der'
del data['cert0']
del data['cert1']
del data['cert2']
del data['key']


# Epson doesn't seem to like bundled certificates,
# so split it into its componens
f = open(CERTFILE, 'r')
full = f.readlines()
certno = 0
certs = dict()
for line in full:
  if not line.strip(): continue
  certs[certno] = certs.get(certno, '') + line
  if 'END CERTIFICATE' in line:
    certno = certno + 1
files = {
  'key': open(KEYFILE, 'rb'),
for certno in certs:
  assert certno < 3
  files[f'cert{certno}'] = io.BytesIO(certs[certno].encode('utf-8'))

# step 3, submit the new cert
r = requests.post(upload_url, cookies=jar,

# step 4, verify the printer accepted the cert and is shutting down
if not 'Shutting down' in r.text:
assert 'Shutting down' in r.text
print('Epson certificate successfully uploaded to printer.')

Update (2020-12-29):

If you are having problems with the Scan to Email feature, with the singularly unhelpful message “Check your network or WiFi connection”, it may be the Epson does not recognize the new Let’s Encrypt R3 CA certificate. You can address this by importing it in the Web UI, under the “Network Security” tab, then “CA Certificate” menu item on the left. The errors I was seeing in my postfix logs were:

Dec 29 13:30:20 zulfiqar mail.info postfix/smtpd[13361]: connect from epson.majid.org[]
Dec 29 13:30:20 zulfiqar mail.info postfix/smtpd[13361]: SSL_accept error from epson.majid.org[]: -1
Dec 29 13:30:20 zulfiqar mail.warn postfix/smtpd[13361]: warning: TLS library problem: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca:ssl/record/rec_layer_s3.c:1543:SSL alert number 48:
Dec 29 13:30:20 zulfiqar mail.info postfix/smtpd[13361]: lost connection after STARTTLS from epson.majid.org[]
Dec 29 13:30:20 zulfiqar mail.info postfix/smtpd[13361]: disconnect from epson.majid.org[] ehlo=1 starttls=0/1 commands=1/2

Update (2021-08-01):

The script was broken due to changes in Let’s Encrypt’s trust path. Seemingly Epson’s software doesn’t like certificates incorporating 3 PEM files and shows the singularly unhelpful error “Invalid File”. I modified the script to split the certificate into its component parts. You may also need to upload the root certificates via the “CA Certificate” link above. I added these and also updated the built-in root certificates to version 02.03 and it seems to work:

  • lets-encrypt-r3-cross-signed.pem 40:01:75:04:83:14:a4:c8:21:8c:84:a9:0c:16:cd:df
  • isrgrootx1.pem 82:10:cf:b0:d2:40:e3:59:44:63:e0:bb:63:82:8b:00
  • lets-encrypt-r3.pem 91:2b:08:4a:cf:0c:18:a7:53:f6:d6:2e:25:a7:5f:5a

They are available from the Let’s Encrypt certificates page.

Apple iCalendar's buggy SNI

TL:DR If you use Apple’s calendar client software, do not run the server on an IP and port shared with any other SSL/TLS services.

I run my own CalDAV calendar server for my family and for myself. For a very long time I used DAViCal, but it’s always been a slight annoyance to set up on Apple devices because they don’t like DAViCal’s https://example.com/davical/caldav.php/majid URLs. What’s more, recent versions of iCalendar would pop up password prompts at random, and after re-entering the password a couple of times (once is not enough), would finally go on and work. The various devices would also all too often get out of sync, sometimes with the inscrutable error:

Server responded with “500” to operation CalDAVAccountRefreshQueueableOperation

requiring deleting the calendar account and recreating it by hand.

I tried replacing DAViCal with Radicale today, with the same flaky user experience, and I finally figured out why: Apple uses at least a couple of daemons to manage calendar and sync, including dataaccessd, accountsd and remindd (also CalendarAgent depending on your OS version). It seems some or all of them do not implement Server Name Indication (SNI) consistently. SNI is the mechanism by which a TLS client indicates what server it is trying to connect to during the TLS handshake, so multiple servers can share the same IP address and port, and is an absolutely vital part of the modern web. For example many servers use Amazon Web Services’ Elastic Load Balancer or CloudFront services, which are used by multiple clients, if Amazon had to dedicate a separate IP address for each, it would break their business model1.

Sometimes, those daemons will not use SNI, which means they will get your default server. In my case, it’s password-protected with a different password than the CalDAV one, which is what triggers the “enter password” dialog. At other times, they will call your CalDAV server with dubious URLs like /.well-known/caldav, /principals/, /dav/principals/, /caldav/v2 and if your server has a different HTTP password for that and sends back a HTTP 401 status code instead of a 404 Not Found, well, that will also trigger a reauthentication prompt.

Big Sur running on my M1 MacBook Air seems to be more consistent about using SNI, but will still poke around on those URLs, triggering the reauthentication prompts.

In other words, the only way to get and Apple-compatible calendar server running reliably is to dedicate an IP and port to it that is not shared with anything else. I only have one IP address at home where the server runs, and I run other vital services behind HTTPS, so I can’t dedicate 443 to a CalDAV server. Fortunately, the configuration will accept the syntax example.org:8443 to use a non-standard port (make sure you use the Advanced option, not Automatic), but this is incredibly sloppy of Apple.

  1. Amazon does in fact have a Legacy Clients Support option, but they charge a $600/month fee for that, and if you need more than two, they will demand written justification before approving your request. ↩︎