Deep packet inspection rears it ugly head

Last Friday I started noticing error messages in my production environment. URLs were being mangled, two consecutive characters being replaced by 0x80 and 0x01 or 0x80 and 0x04, causing UTF-8 decode exceptions to be logged, as well as failures for the cryptographic hash function we use to secure our URLs. As a general principle, I take any such unexpected exceptions very seriously and started investigating them, one concern being that some of our custom C extensions to nginx could be responsible for data corruption under heavy load.

I ran snoop (a Solaris utility similar to tcpdump) on one of our production servers, and after combing through 180MB of packet traces with Wireshark, it turned out the data was being corrupted before even hitting our web servers. While it was a relief to find out our own infrastructure was not to blame, I still had to identify the culprit, e.g. whether our hosting provider’s switches, firewalls or load-balancers were to blame.

TCP has built-in checksums, so a malfunctioning switch working at layers 1–3 would not cause this problem, a corrupted packet would be dropped and resent, with a slight hit on performance but no errors. Thus the problem would need to be at a L4 or higher device such as a load balancer.

I added some extra logging and let it run over the weekend. After analyzing the data, it turns out the problem is very circumscribed (76 requests out of hundreds of millions), and all the affected IP addresses come from the same ISP, Singapore Telecom Magix (AS9506). The only plausible explanation is that SingTel is running some sort of deep packet inspection gear, and some of the DPI gateways have corrupt memory or software bugs, that are causing the data flowing through them to get corrupted,

Deep Packet Inspection is a scourge the general public is insufficiently aware of. At a high level, DPI gateways watch over your shoulder as you use the Internet. They decode the data packets passing through them, reconstruct unencrypted HTTP requests (in other words, spy on your browsing history). In their transparent proxy incarnation, they can rewrite the requests or responses. Verizon Wireless uses the technology to resize and recompress images or videos requested by smartphones. Back when I used to work for France Telecom (circa 1996-1999), vendors would regularly approach us to peddle their wares and how they would allow us to price-gouge our customers more effectively. Hardware has progressed dramatically since and a single Xeon processor is capable of inspecting at least 10 Gbps of data.

The whole premise of DPI and other snooping devices is profoundly repugnant to me as a former network engineer, on both moral and technical grounds. Any additional “bump in the wire” slows things down and is yet another potential point of failure, as shown by this incident, but the potential for abuse is the real concern. Not to mince words, the legitimate purposes for the technology, such as fighting cybercrime, are just rationalizations, it was really developed for purposes most people would consider abusive.

When I joined FT, I had to go to a Paris courthouse and swear a solemn oath to defend the privacy of our customers’ communications, and report any infringement of the same. DPI technology originates in spy agencies, and is much beloved of authoritarian governments. China uses the technology, combined with voice recognition, to drop calls at the merest mention of the word “protest”. The Ben Ali regime in Tunisia used it to snoop Facebook users’ authentication cookies. Singapore’s government has a well-demonstrated intolerance of criticism, and who knows what SingTel is doing with their defective gear? Western companies like Cisco were disgracefully eager to sell censorware to dictatorships, but those governments now have homegrown capabilities from the likes of Huawei.

For telco oligopolies, the endgame is to practice perfect price discrimination, e.g. charge you more for packets that carry a voice over IP call or a Netflix video on demand session that compete with the carriers’ own services. Telcos and cablecos cannot be permitted to use their stranglehold over public networks for what is essentially racketeering. Strowger invented the automatic telephone switch because the operator at his manual exchange would divert his calls to one of his competitors, her husband. Telcos, in their monopolistic arrogance, feel a sense of entitlement to all the value the network creates, even when they are not responsible, and want to reverse this. Letting them get away with it, as is consistently the case in the US, is a recipe for long-term economic stagnation.

What can we as the general public do to fight back? The telcos are one of the largest lobbies in Washington, and wireless spectrum auction fees are one of the crutches propping up Western budgets, so help is unlikely to come from the venal legislatures. The most practical option is to start using SSL and DNSSEC for everything. Google now offers an encrypted search option and Facebook has an option to use SSL for the entire session, not just for login.

Update (2012-10-16):

It seems Verizon also uses DPI to build marketing profiles on its users, i.e. categorizes you based on your browsing history and sells you to marketers. You can opt out, but the practice is deeply worrisome.

RapidSSL 1 – GoDaddy 0

My new company’s website uses SSL. I ordered an “extended validation” certificate from GoDaddy, instead of my usual CA, RapidSSL/GeoTrust, because GoDaddy’s EV certificates were cheap. EV certificates are security theater more than anything else, I probably should not have bothered.

Immediately after switching from my earlier “snake oil” self-signed test certificate to the production certificate, I saw SSL errors on Google Chrome for Mac and Safari for Mac, i.e. the two browsers that use OS X’s built-in crypto and certificate store. I suppose I should have tested the certificate on another server before going live, but I trusted GoDaddy (they are my DNS registrars, and competent, if garish).

Big mistake.

I called their tech support hotline, which is incredibly grating because of the verbose phone tree that keeps trying to push add-ons (I guess it is consistent with the monstrosity that is their home page).

After a while, I got a first-level tech. He asked whether I saw the certificate error on Google Chrome for Windows. At that point, I was irate enough to use a four-letter word. Our customers are Android mobile app developers. A significant chunk of them use Macs, and almost none (less than 5%) use IE, so know-nothing “All the world is IE” demographics are not exactly applicable.

After about half an hour of getting the run-around and escalating to level 2, with my business partner Michael getting progressively more anxious in the background, the level 1 CSR tells me the level 2 one can’t reproduce the problem (I reproduced it on three different Macs in two different locations). I gave them an ultimatum: fix it within 10 minutes or I would switch. At this point, the L1 CSR told me he had exhausted all his options, but I could call their “RA” department, and offered to switch me. Inevitably, the call transfer failed.

I dialed their SSL number, and in parallel started the certificate application process on RapidSSL. They offered a free competitive upgrade, I tried it, and within 3 minutes I had my fresh new, and functional certificate, valid for 3 years, all for free and in less time than it takes to listen to GoDaddy’s obnoxious phone tree (all about “we pride ourselves in customer service” and other Orwellian corporate babble).

I then called GoDaddy’s billing department to get a refund. Surprisingly, the process was very fast and smooth. I guess it is well-trod.

The moral of the story: GoDaddy—bad. RapidSSL—good.

Update (2012-08-26)

I switched my DNS business from GoDaddy to in December 2011 after Bob Parsons’ despicable elephant-hunting stunt.

Clueless SaaS providers can leave you with egg on your face

While cleaning out my spam folders, I noticed a disturbing trend: a number of the spam were sent to vendor-specific email addresses I had set up to communicate with Parallels, Joyent and Shoeboxed. As a security measure, I do not give my personal email address to vendors, only aliases. The email address I used in the past for Dell was, for instance (I now use a different domain). A few years back, I started receiving pornographic spam at that address, which led me to think either Dell had secretly adopted a radically new diversification plan, or that their customer database had been compromised. Needless to say, this did not reflect well on Dell. I canceled that alias and stopped dealing with Dell.

I contacted the support for the three vendors. Joyent got back to me, and said:

We have traced this back to a third-party provider that was used to distribute service notifications. We have been in contact with this service provider, and they have determined that subscriber email addresses of their clients were compromised. They have launched their own investigation, which is ongoing, and have also reached out to their local FBI office.

After some digging, I found some interesting posts. Some email marketing company called iContact, that I had never heard about before, was the source of the compromise. They claim to be SAS-70 compliant, but of course like most bureaucratic certifications, SAS-70 is mostly security theater that makes sysadmins’ life miserable for no meaningful security benefit (SAS-70 auditors, on the other hand, profit handsomely).

Just another example of how outsourcing critical functions to outside vendors can backfire spectacularly and take down your own reputation in the process.

Broken SPF records

I have SPF verification enabled on my mail server. While SPF is no panacea for the problem of spam, it is quite effective at ensuring spammers do not forge the sending address to impersonate someone else, and cause some poor innocent soul to receive in a boomerang effect the torrent of complaints hurled at them.

Unfortunately far too many lame organizations (cough, Google) qualify their SPF record using a too permissive ?all or ~all clause, which means they have servers other than those listed, and thus their SPF record is useless for filtering purposes.

In the last month, I noticed the opposite problem: I did not receive emails from Eurostar and BookMooch because their SPF records did not list the mail servers they actually use. If they are not clueful enough to manage a simple list of IP addresses, or have basic change management discipline, they should do us all a favor and ditch the SPF record they clearly are incapable of maintaining.

Fie on parasitic US cellcos

The Economist has an excellent article on how Indian mobile phone companies cut costs. They have an ARPU of $6.50 a month yet operate with a 40% gross margin. If US cellcos were run as efficiently, they would have a 1200% gross margin on the $51 monthly ARPU!

The time has long come to stop coddling grossly inefficient and anti-competitive cellular carriers in the West. They are no longer fledgling businesses in the shadow of landlines, quite the opposite, in fact. One good place to start would be to require them to offer consumers the choice of carrier for international calls and for roaming, as is the case with landlines. Their rates are simply extortionate.

At long last real broadband in San Francisco

I upgraded my broadband connection yesterday from a puny 3-6Mbps/384-768K DSL connection to 20Mbps symmetrical Metro Ethernet service from an outfit called WebPass. My current ISP, Raw Bandwidth, has excellent service with no restrictions on hosting servers or traffic shaping shenanigans unlike the likes of Comcast, but they are still hobbled by the AT&T last-mile connection.

WebPass finesses around the incumbent monopoly by using newer buildings’ data-grade wiring plant to bring 100MBps Ethernet connections right into your home (all they had to do was change a wall plate and patch some cables in the closet) and use microwave links to backhaul traffic to their data center. They claim to use a mesh network for backhaul, but I think this just means a standard network of microwave links where some sites have to hop multiple microwave links to get to the transit connection, rather than a purely centralized hub and spoke model. In my case their offices are a mere two blocks away. This would allow me the pleasure of ditching the scumbags at AT&T altogether (were it not for the fact my building requires an entirely unnecessary landline for its security system).

AT&T is probably the worst telco in the US now, and is notorious for starving its infrastructure of investment to maximize short-term profits, unlike Verizon, who is investing heavily in its FiOS optical network. Unfortunately San Francisco is in AT&T territory and will not get true optical networks anytime soon. Municipalities can usually reassign the cable franchise every so many years, but there is no such provision for involuntary transfer of telcos that I know of.

The new service is $45 a month with no installation fee, vs. $70 a month for Raw Bandwidth, but it does not include a static IP address (they do offer it as part of their prohibitively expensive metered business service). Configuring my home router (a Cisco 877) to use both connections was incredibly painful, but I will run the two ISPs side by side for the next few months. If WebPass proves as reliable as Raw bandiwdth, I may just have to find a work-around for the static IP issue, or just rely on DHCP lease pinning.

If you live in San Francisco, or are moving there, definitely have a look at the buildings they have covered. The service is a glimpse of what people not in broadband backwater USA get.

Today is a great day for the Internet in France

The content producers’ lobby is very ancient and powerful in France (it was started by the playwright Beaumarchais in the 18th century). The fact President Sarkozy’s wife is an important rights holder may have something to do with his determination to pass the abject Hadopi law, which would cause Internet users caught illegally downloading content to be cut off from the Internet (while still having to pay their ISP fees).

The law was exceedingly stacked towards the content industry. The burden of proof was on the defendant rather than the prosecution, and an extra-judicial quango named Hadopi was to be set up to enforce these sanctions. The European Parliament, to its credit, had opposed such measures and restated that Internet access is a fundamental right that can only be curtailed by proper judicial authority. The first reading of the law led to a surprise defeat, as the majority UMP legislators were unenthusiastic about supporting a law that would alienate the young, and absenteeism was such that the minority Socialist party managed to overwhelm those few present. This is one of the exceedingly few times I actually agree with the feckless Socialists… The President brought his whip to bear and the law was put back on the agenda and voted in the second time.

Today, the Conseil Constitutionnel ruled on a challenge to the law put by Socialist parliamentarians, and gutted it in line with the European Parliament. In doing so, it affirmed that Internet access is a fundamental human right, drawing all the way back to the original Human Rights declaration of 1789, and that Internet users are innocent until proven guilty.

This is an important decision. In Roman law, judges’ discretion is much more limited than in the Anglo-Saxon Common law tradition. The US Supreme Court found in Roe vs. Wade a right to abortion in the US Constitution that is far from obvious, and such a decision by unelected judges lacked universal legitimacy. In contrast, abortion was legalized by an act of Parliament in France, which is why opposition to it is nowhere near as bitter as in the US. The Conseil Constitutionnel does not invent constitutional principles, it only censures laws or more commonly individual articles (the role of ultimate court of appeals belongs to another institution, the Cour de Cassation). The significance of it finding Internet access a fundamental right cannot be overstated.

SOCKS and SSH, two great flavors that go together

I am currently in New Orleans for a friend’s wedding, and staying at the InterContinental. The hotel has wired Internet access, but like all expensive hotels, wants to charge an extortionate fee ($7/hour) for it. This is annoying as the same hotel chains’ budget-priced hotels usually offer it as a complimentary service.

I noticed my email was going through, however. On further inspection, it turns out they only intercept port 80 HTTP traffic, but not on other ports. Security through (very thin) obscurity, in other words.

I tried using Firefox from my home machine over X and SSH port forwarding, but it was painfully slow.

At this point, I was considering setting up a HTTP proxy on my home machine and using it over SSH port forwarding, but I remembered reading something about SSH and SOCKS. I had never used a SOCKS proxy before, but it turns out this is incredibly easy: just add the -D option to ssh with a local port number, e.g. 9999, and set up your browser to use localhost:9999 as the SOCKS proxy. It worked flawlessly with my Mac OS X SSH client and Solaris 10 stock server.

This has applications beyond routing around hotel paywalls. Many public WiFi access points are unsecured. Even if they are legit (many are peer-to-peer vs. infrastructure, and presumably used by thieves to harvest passwords), they can be snooped for passwords trivially easily. Using SSH and SOCKS provides you with security when using an untrusted Internet access point, even for non-SSL sites. My email uses IMAPS and SMTP TLS so I don’t need to reconfigure it to use SOCKS, but that would also be an important protocol to secure.

All in all, a very worthwhile addition to my toolset. I can’t believe I waited so long to try it.

Update (2009-04-12):

To its credit, New Orleans’ Louis Armstrong international airport has free WiFi throughout the terminal. Chic!

Whither IP-based home automation?

Home automation units based on X10/Insteon or proprietary systems like Control4 or Savant start at $100-200. At a time when you can buy a fully functional WiFi router with a 200+MHz processor, a minimum 8M of RAM, 16MB of flash for under $50, why is there not a home automation system that costs $50 and uses standard TCP/IP and WiFi for connectivity?

Another reason why I build software from source myself

Some yahoo at Debian found what he thought was a bug in OpenSSL, and decided to comment out some code without having any clue what purpose it served. That purpose was to seed a pseudo-random number generator with entropy from memory, specifically /dev/random. This only broke the cryptographic security of OpenSSL on Debian (and thus Ubuntu) while being mostly undetectable. It’s quite likely attacks of the same ilk were deliberately planted by various spy agencies.

This is just an extreme example of why I prefer to build open-source software from source code myself rather than trust blindly in some packager whose choice of compile-time settings almost certainly doesn’t match mine. I have a framework of makefiles that specify how each package is built from source (meta-makefiles, really). This includes checking for new versions of the package, setting configure options and make environment variables. For instance, to fetch the most recent version of OpenSSL, all I do is make sync-openssl; make openssl then as root run make install-openssl. The maintenance burden is low as I have been assembling these metamakefiles over the last 12 years, targeting Solaris and OS X. The end-result is a deterministic build according to my specifications.

My process would not ward against a malicious attack like Brian Kernighan’s notorious trusting trust attack, but it has served me well over the years.

Ethernet – accept no imitations

While reading an article about Brocade/Foundry’s product plans, I learned about Convergent Enhanced Ethernet (CEE), also known as “Data Center Ethernet”. As if Ethernet needed blessing from the price-gouging storage vendor community to enter the data center…

CEE sounds like just one more in a long line of failed “standards” like IsoENET that take Ethernet, find some supposed nit to pick with it, and add all sorts of baggage to address hypothetical requirements. In the case of IsoENET, it was jitter, and in the case of CEE, it is packet loss, never mind that function belongs to TCP at layer 4, not Ethernet at layer 2. It is a predictable result of the Fibre Channel community’s lame attempts to head off iSCSI by putting FCP directly over Ethernet (FCoE).

This basically reflects an attitude of “my traffic is so special, it can’t be allowed to mingle with unwashed Ethernet packets”, and reminds me of how France Telecom Labs circa 1996 was very proud to show a prototype of “World Wide Web without requiring the use of IP”, as if the fact the web uses IP rather than ATM was a barrier to adoption, rather than a success factor…

The reason why Ethernet is so phenomenally successful is that it is simple, easy to implement and cheap. Any attempts to add complexity will only delay time to market, limit economies of scale and add cost, until whatever comes out becomes just as expensive as the Fibre Channel ports and adapters the whole world is trying to ditch in favor of fast, inexpensive vanilla Ethernet. Then again vendors like Brocade grew fat on gouging Fibre Channel customers and hope they can reverse the trend of commodification and keep on with their little racket.

It’s just not going to happen, specially in a tough economic environment where IT expenditures are contracting and any attempt by vendors to foist their overpriced proprietary dead-end marketectures will be treated harshly by buyers.

Unbound from djbdns

I am experimenting with IPv6 at home using Hurricane Electric’s free tunnel broker. I had to upgrade my Cisco 877 router’s RAM, flash and software to get IPv6 support, and also my local caching DNS resolver, dnscache. There are IPv6 patches for djbdns, but since I installed them my DNS lookups seem slow. Using snoop and ethereal, it looks like the behavior of the server with or without the patches is quite different.

Considering the fact that djbdns has not had an official update since 2001, only collections of patches from third-parties, it was time to change, even though it was immune by construction to the Kaminsky bug. I opted for unbound from the same people who wrote the high-performance NSD server used on the RIPE root nameserver. It has a relatively simple architecture design for performance and security, and it supports DNSSEC, something that will become increasingly important.

While the configuration file format for unbound is simple, unlike the nightmare that is BIND, the devil in the details made the migration more painful than it ought have been, thanks in part to my split-horizon DNS configuration for machines on my local subnet. I don’t know if it is placebo effect, but my queries now feel faster.

US banks lag behind in secure email adoption

My banks send me monthly reminders when a statement is ready, but I have to log onto their site to actually get it. This is quite annoying, I would much rather have them simply attach the statements to the notification emails, but I can understand their security concerns. The current system does encourage bad habits that can be exploited by phishers, however.

One of my colleagues informed me that in Japan, banks will actually send them by email using S/MIME public key encryption. I have a S/MIME certificate courtesy of the Thawte web of trust (in fact I am also a Thawte WOT notary) but no US bank that I know of supports this. Secure email adoption is so low in no small part due to the NSA’s successful campaign to make encryption inconvenient to obtain. All major email clients support it (Outlook, Apple, Thunderbird, and so on), but webmail users don’t even have the option. This is just another illustration of how the US is lagging behind Asia and Europe in Internet adoption.