Public Key Authentication

I haven't been updating this blog lately because I've been trying to get my webserver set up first, so that I could put the pictures there and just link to them from the blog posts. (Uploading pictures to this blog is a tedious process, so I'm planning to create a website that I can just drop a folder full of pictures onto and have them all display nicely. I suppose I could just get a Flicker account instead, but I like doing things the hard way.)

So that I won't have to type my password over and over again to manage my webserver, I took an evening to set up public/private key authentication so I can just use that to prove to my webserver who I am when I log in, and then I don't need a password. Well that was the plan. It ended up taking an evening, then a weekend, then another few evenings, then a week... Every time I thought I had everything set up properly, the next time I came back and tried to log in, the key didn't work. Eventually it got to the point where I'd make a key, it would work exactly one time, and then never work again until I made a new key. (If you know how public/private keys work, this makes NO SENSE. But there it was.)

I spent more time troubleshooting this problem than all the cumulative time I will save from not having to type a password in the future. Nothing wastes your time like a technology designed to save you time, right? But I'm stubborn and the longer this went on the more determined I got to figure it out. See, I have a theory about life: success or failure is self-reinforcing between your beliefs and your actions. If you believe you can't make technology work for you, then when faced with a problem you'll give up too soon. By failing you proved right your own conviction that you would fail. Because you'll never know if the right solution was miles away from what you attempted, or whether it would have been just around the corner. On the other hand if you maintain an unshakeable belief that eventually, something you try will work, eventually it does, and you vindicate the confidence you had in the first place.

Turned out the problem in this case stemmed from an innocent decision I made when installing Ubuntu Linux on the server. In this new version of Ubuntu, they added a feature for encrypting your home directory. The install program asked me if I want to do this, and explained that it would be perfectly transparent to me; the operating system would magically decrypt and mount the home directory for me when I log in, and safely unmount it when I log off. Totally convenient - I'd never notice anything different. "Why not?" I thought. It didn't sound like much trouble and who wouldn't want more security on their webserver? So I left it enabled. BIG MISTAKE.

I would never have guessed there would be an interaction between this feature and using public/private key logins until I was looking at the ssh logs to try to figure out why I couldn't log in with my key. At first I thought I had the wrong log - why were there messages from the home directory encryption service mixed in with my ssh logs? And why was it not finding the key sometimes? Wait wait the order of those operations doesn't look right - shouldn't it be decrypting the home directory before looking in it for the key? Ahhhhh... Dammit. Now I get it.

Here's what was going on: the way public/private key ssh logins work is the server looks in your home directory for your public key to use to recognize you. But your home directory is encrypted. It gets decrypted for you when you log in. But you can't log in until it uses the public key to authenticate you, which it can't pull out of the home directory because it's still encrypted. "There's a hole in my bucket, dear Liza, dear Liza..."

You can verify if you've got the same problem as I did: as long as you're logged into the server in one terminal window, public/private key authentication will work in a 2nd terminal window. Log out of all the sessions, and now you can't authenticate anymore with your key. Log back in, and now your key works again in other terminals too. If this works, send an angry email to Canonical. Tell them I sent you.

The moral of the story is don't use the encrypted home directory feature, it was released half-baked. Have you heard the expression "Beware Greeks bearing gifts?" The meta-moral here is beware operating systems bearing gifts of "features" you didn't need!