About once a week, I get an email in my mailbox that reads like this:

Hey, Kiln looks neat, but Git is totally the bee’s knees, so why the fuck are you using Mercurial?

Note that these emails are rarely (if ever) actually interested in why Kiln chose Mercurial; what they’re instead interested in is trying to piss me off enough that I get into a flamewar about why Mercurial is going to bring about Nirvana while Git causes people to eat babies using nothing but A1 sauce and a spork.

This is stupid.

Mercurial and Git are both DAG-based DVCSes. They use the same patch format. They both handle directories implicitly. They both can autodetect file deletions and renames. They both run on just about every platform I can think of. They both do a bunch of “cool” stuff, like rebasing, editing history, signing changesets, and serving as the inspiration for Ferris Bueller. They both have nearly identical performance characteristics, they both have social code sharing sites, they both are used by really big projects, they both have really good documentation, and they both got compared to James Bond or MacGyver or something like that in an analogy I didn’t really follow.

So why is there so much hating? I think what’s going on is that people are coming to these tools from Subversion or CVS, have their massive epiphany on how totally awesome DVCSes are, and then assume that only their tool can have this level of awesome, so they begin evangelizing. The problem, of course, is that the other guy feels the same way, and is also evangelizing, so the Git and the Mercurial guy end up in a locker-room-style temper-tantrum over whose tool has the best performance or whatnot, instead of how much more awesome their tools are than the competition.

This has to stop.

Mercurial’s enemy is not Git. Git’s enemy is not Mercurial.

Their enemy is Subversion.

For example, take a look at this tripe. Anyone who has seriously used Mercurial or Git for any length of time is going to spend most of the video alternating between laughing, peeing their pants, and trying to explain that they merely spilled a bunch of water on their crotch, honest. While there’s some truth to the video—a lot of people do ditch Subversion for off-line commits or for shelving or the like—the video also totally misses why people love and stay with DVCSes, which is that they make branching and merging actually work like they’re supposed to, and make source control so fast and seamless that you find you’re suddenly using it for everything.

But if you show that video to a Subversion user, they’re going to nod. And there’s really no reason for them not to: if you don’t grok what the DAG gives you, if you’ve never kicked an entire branch back-and-forth across a LAN, if you’ve never used a site like Bitbucket or GitHub, then the only tangible benefits you see from DVCSes are…well, partial commits, and the fact that their “checkouts” aren’t littered with .svn directories all over the place. And these are problems that Subversion’s upcoming versions actually might solve.

So it’s time to focus on the real enemy: the holdouts still using centralized systems. The way I see it, there are three parts to this:

1. Git and Mercurial need to do a better job handling one thing that Subversion is still better at: binary files. I know there are Git projects that are working on improving the transfer and storage of binary files within Git’s existing repository format, such as git-bigfiles. Mercurial is taking a slightly different tack through projects such as bfiles, which aim to deliberately move large binaries out of the store. We need to improve these workflows so that “DVCSes are totally better, unless” becomes simply “DVCSes are totally better.” (For our part, Fog Creek is helping fund development of bfiles via UCOSP, a semester-long student project.)
2. Git and Mercurial advocates need to remember that they need to be converting Subversion, CVS, and Perforce users, not each other. The fundamental evil here is centralized, branchless version control systems that effectively encourage all development to occur in trunk, and which make propagating bug fixes and features properly borderline impossible for all but the most disciplined shops.
3. Git and Mercurial advocates need to remember that anyone going to either system is a win for the both communities. It’s easy, in the yin/yang of Hacker News and proggit, to forget that most developers are not even aware of what DVCSes are or what they do. Yeah. Sounds crazy, I know, but trust me on this. The goal right now, if you honestly believe that the DVCS workflow is better—and I do—should be to get the mindset out there, to make more people aware of what DVCSes have to offer and why they should be using them. I for one definitely do not care whether you end up deciding that Mercurial is better than Git for you or not (well, I kind of do, because I want you to use Kiln, but otherwise…), but I get a warm fuzzy feeling knowing that, if I ever have to work with your code, I’ll be able to use a sane version control tool.

So let’s do it. No more fighting. Git, I hereby acknowledge you rock. And Mercurial, you rock so much I helped build an entire product around you. You’re both awesome. So you two shake hands…very nice. Now, see that other dude over there? The one with no real tagging or branching support?

Let’s get him.

This article is a continuation of Kiln’s Evolution, Part 1: DVCS as Code Review.

In the fall of 2008, Joel was getting increasingly adamant that FogBugz needed source control integration, and most people in the company seemed to think Subversion would probably be the best SCM to make that happen. Tyler and I disagreed, believing strongly that we should use a DVCS instead, and that our code review tool gave a really compelling example of why DVCS was better that any software shop would instantly “get.” But to convince the rest of the company, we’d have to show them a version of our tool that was more polished and usable than what we’d submitted to Django Dash.

And so we began a skunkworks project.

Unable to use time at work on much besides Copilot, I instead used my week of Thanksgiving vacation cleaning up the prototype’s user interface and functionality, named the result Kiln, and gave it its first logo. Tyler spent evenings in December making Kiln a proper, pluggable Django application, made the UI actually usable, and fixed a pile of bugs that would have blown up in our face if we’d tried showing Kiln to anyone else. By January, 2009, Kiln was ready to demo.

Kiln’s interface, directly after the winter skunkworks changes.

After lunch on a cold winter day, Tyler and I dragged everyone out into the kitchen and demoed the current state of Kiln. We showed repository management and the FogBugz-inspired code review workflow, and then made the case that this, or something very similar, should be FogBugz’ source control system.

And an amazing thing happened: somehow, everybody basically agreed. Sure, some people thought Kiln should be in C# or Wasabi instead of Django, no one could agree on whether Kiln should be a direct part of FogBugz or be an independent product, and Tyler and I argued strongly for Kiln as a hosted-only solution to a bunch of people who knew their bread-and-butter came from licensed applications, but everyone agreed that the basic of idea of a Mercurial-powered SCM with DVCS-backed code review made for a compelling product. And so Kiln was born.

For Kiln to develop into an adult, though, we had to assemble a Kiln team. Tyler and I were still working on Copilot, and the newly appointed team lead, Ben Kamens, was busy with the FogBugz 7 release. Even if all three of us started work immediately, we couldn’t possibly turn the project from prototype to beta by our target date of August 2009, and starting immediately seemed…well, optimistic, at best.

But we work at Fog Creek, and if there’s one thing Fog Creek knows how to do, it’s how to help interns churn out awesome products over the course of a single summer. After all, Tyler and I started at Fog Creek by developing all of Copilot in the summer of 2005; why not go for broke and try for a repeat? What we therefore decided to do was to bet the farm and put all of our summer interns on Kiln. The three of us would try to wrap up the work we had to do on our current projects as quickly as possible, and, as we transitioned off, we’d focus purely on building up enough Kiln infrastructure that the interns could immediately be productive when they arrived. Meanwhile, until the three of us could start work on Kiln, we’d have our project managers figure out the details of the user experience so that, once we finally could work on Kiln, we’d be able to focus as much as possible on coding instead of decision-making meetings.

As we moved ever closer to June, we made several key decisions about the design of the product:

1. Kiln could launch hosted-only, but we’d need to ensure that its design was amenable to on-site installation.
2. Kiln would depend on FogBugz for user management and bug-tracking integration, but would otherwise be its own code base.
3. Kiln’s website would be written in C# and ASP.NET MVC, completely freeing it from the FogBugz legacy code base.
4. The part of Kiln that needed to talk directly to the DVCS would be a separate component so that we target different (or even multiple) SCMs without changing the website.
5. Code reviews on branches would be eliminated in favor of arbitrary discussions on files and changesets.

By the time the first interns arrived, we had a beautiful set of specs with lovely Balsamiq Mockups put together by Jason and Dan, and we’d managed to cobble together a basic framework that supported repository hosting and FogBugz integration, and that learned as much as possible from our best example of a known-good ASP.NET MVC code base, StackOverflow.

Our plan paid off ridiculously quickly. A week into the internship, the interns had already managed to get key pieces of Kiln limping along. By the end of the second week, they had enough ownership they were starting to challenge us when they felt the user specs or the engineering didn’t make sense. Their strong focus on core Kiln freed Tyler, Ben and me to focus on performance, billing, On Demand integration, and all the other things that absolutely must get done for a real product, but that no one would otherwise ever do.

Just over a month into the summer, Kiln might not win any speed or beauty awards, but nearly all of its features were working in one way or another, and it was usable for its intended purpose. In other words, we’d hit pre-alpha. With great fanfare, we decided that Kiln was ready for dogfooding, and Kiln development moved to Kiln itself.

There’s a slightly unfortunate thing about dogfooding, though: features that looked great on paper, and even worked perfectly in the prototype, end up not being what you want in the real product. Some interfaces end up not scaling the way you want. Some end up too complicated, or end up solving one particular problem at the expense of all others. It’s a testament to our PMs that we had comparably few of these occur, but sometimes the difference between the prototype and what we ended up shipping was massive. For example, compare the Balsamiq mockup of the code review system, which was actually used by the pre-alpha:

with the version that we ended up actually shipping:

Or take a look at the original specification for the Kiln Dashboard:

compared to the shipping equivalent, the Activity Feed:

(I apologize about using the mockups, rather than screenshots, for the earlier versions; trying to get Kiln circa June 2009 running at this point proved a royal pain in the butt, and I don’t honestly think that it makes a big difference.)

What’s not obvious in these two screenshots is that the change from the pre-alpha interface to the shipping interface frequently happened over the course of just a couple of weeks. Everyone was very vocal about what they liked and didn’t like, and the interns were happy to go through several iterations rapid-fire to find one that everyone liked. In that way, the weakest parts of Kiln ended up getting the most attention, and rapidly matured into some of its strongest features.

In a massive code sprint at the end of the summer, Kiln matured into something resembling a fully grown product. One of our interns made a beautiful JavaScript renderer for Kiln’s DAG, the novel repository management our PMs designed fully matured into beautiful JavaScripty goodness, our FogBugz/Kiln workflow became increasingly seamless, and we further loosened review requirements so that you could review arbitrary discontiguous changesets. We transitioned Copilot and FogBugz to be hosted on Kiln as well, got mostly positive feedback from the rest of the team, and worked on swiftly addressing their complaints. Despite these feature additions, Kiln’s performance went from tolerable, to better, to fast. We knew we had a winner on our hands. We prepared the FogBugz On Demand environment to become Kiln On Demand, made our first deployment, and turned the switch for our first batch of beta users.

And while you might expect this part of the story to be about everything going haywire and all hell breaking loose, what actually happened is that, against all odds, everything basically worked. The beta was quite boring: while there were a lot of bugs to fix at first, and some of them were extremely tough (for example, supporting very large repositories, or making history views faster, or legitimately supporting Internet Explorer) or really ticked off our customers (Kiln at once point let you rename and move repositories without breaking URLs, which sounded like an absolutely great idea when I helped hammer it through the design committee, but which completely blew up in one of our client’s faces a few weeks later), everything basically worked. Our beta testers seemed increasingly excited about the product. All of our gambles seemed to have paid off as we readied Kiln 1.0 for its November launch date.

Except, of course, that Kiln did not ship in November, 2009. That’s because, just a week or two before it was supposed to ship, we went to the Business of Software conference in San Francisco.

You see, that was where we realized we were doing it all wrong.

To be continued…

As Kiln draws ever closer to release, I realized that we have long since passed the point where I should move all of my personal projects to it.

So as of today, I have.

If you’re interested in grabbing the most recent version of FogBugz Middleware, my fork of Kiln Backup, or any of my other public projects, they’re now all at https://bqb.kilnhg.com. Log-in with the user guest and the password anonymous, and you’ll have full read-only access to the entire site.

Even if you’re not that interested in the stuff I’ve written, you still may be interested in checking out the site: by enabling read-only guest logins on my FogBugz account, I’ve made it very easy for you to take a look around a real, active Kiln and FogBugz install, without setting things up or filling out any annoying forms. If you just wanted to a get a quick feel for what Kiln looked and felt like, now’s your chance.

So go ahead and check it out. Feel free to post question or comments here, or (if appropriate) over at the Kiln StackExchange, and I’ll be happy to answer them.

Google’s motto is, “Don’t be evil.”

I’ve always found that motto disturbing for two reasons. First, a company that can differentiate itself—successfully, no less—from its competitors merely by promising not to be evil implies that the average company is ridiculously corrupt. A person who announced, “My motto is, ‘don’t shoot people’” would be notable because no one thinks you should shoot people, making the promise weird and redundant—not because the promise represented some great sacrifice. Yet Google’s promise to do no evil somehow hits people, especially those in the tech industry with fresh memories of Microsoft in the 90s and the specter of Oracle in the 2000s, as a breath of fresh air. Great for Google, but pathetic for our industry.

But the second reason, and the more important one for me, is that “Don’t be evil” is not the same as “Do the right thing.” A person who watches idly while a bully beats someone up isn’t being evil, but they are being a coward, and they are not doing the right thing. Their interference could save a poor victim a world of pain and suffering, probably at minimal risk. Instead, they simply watch the bully, knowing that they themselves would not do the same thing. This may not be doing evil, but it’s also not the moral high ground. Knowing you would never beat someone up is not the same as protecting those weaker than you.

Google chose its motto carefully; if its motto were instead, “Do the right thing,” then it would have no presence in China. For all the corruption that people accuse our government of perpetrating, our government does not censor the Internet, does not shoot and incarcerate those who disagree with it, does not deny its citizens the right to vote, and does not persecute religious minorities as a matter of state policy. China does. And until today, while Google may not have been evil in China, they certainly enabled evil to go about its business by running a censored search engine there. They were unequivocally better than Yahoo, who handed over the names and email addresses of dissidents, but they weren’t doing the right thing, either. They weren’t standing up to an autocratic, dictatorial regime.

As of today, that has changed. Google has announced that they will no longer censor their Chinese search results. While you could argue that Google’s doing this out of anger that their resources have been hacked, rather than out of a genuine desire to protect its users, their result of their actions is beyond dispute: they are taking the moral high ground. And potentially at great cost: while China has certainly failed to materialize as the unstoppable threat to the West that pundits were claiming it would become two decades ago, it’s nevertheless home for nearly a billion people, and shows no sign of stopping its economic growth in the near future. For Google to make a move that will almost certainly sacrifice any chance they have of winning the Chinese market is an economically painful move.

But it’s the right move.

So, at least for today, at least this once, look at Google as a company which is not merely avoiding perpetrating evil. Google is doing the right thing, at great cost. And they deserve to be lauded for that.

Microsoft and Yahoo: this is your turn to follow in Google’s footsteps. Do the right thing. It won’t make you money. In fact, it’ll cost you. But it’s the right thing to do.

Google: where it’s not don’t be evil. It’s, “Do the right thing.”

Congratulations.

Whenever I browse the Android Marketplace, I’m utterly amazed by how many “app reviews” are nothing but spam. The problem is so pandemic that I have to conclude that Google has thus far done absolutely nothing to combat the problem. Shopping in the Marketplace ends up feeling like going through a dirty bazaar, surrounded by panhandlers and con artists looking to make a cheap buck. I don’t care how good the deals may be; if shopping ends up being an annoying experience that makes me feel dirty, I’m unlikely to bother going in the first place. In the Marketplace, that translates to finding nothing but crap reviews, making shopping for any given application basically a crap shoot. Plus, the apps themselves end up looking like schlock you’d find on a spyware site.

If Google’s serious about trying to compete with the iPhone App Store, they need to get off their feet and fix this problem right now.

Well. At least it makes it suck less.

I bought a Droid the day it came out. While it was a tremendous improvement over my BlackBerry, I’ve been disappointed with the phone overall. The battery cover comes off constantly ([2], [3]), the phone’s proximity sensor was extraordinarily finicky (usually resulting in me hitting the “mute” button with my cheek in the middle of a call), the camera was all but useless, and, for reasons I did not really understand, my Android developer phone running Android 1.6 provided a much smoother user experience than the vastly-more-powerful-on-paper Droid. In other words, the Droid was a solid upgrade from what I had, but still disappointing. I have to agree with Dave Winer’s now-famous rant on why the Droid sucks.

Last night, Motorola and Google unleashed Android 2.0.1 as an over-the-air update. While the update does little about the battery cover, it seems, at least so far, to resolve nearly all of the software issues. The proximity sensor’s logic seems improved, though not perfect; many operations are visually smoother (although oddly still not universally as smooth as the G1); the camera’s usable taking pictures, rather than mocking the incompetence of Motorola’s engineers; and there have even been some very nice visual refinements to fonts and color schemes. Best of all, and unusual for the first update to a new device, nothing broke: sudoku, SpacePhysics, Twidroid, TripIt, and other applications seem to still be working just fine.

So if you were previously hesitant about buying a Droid for software reasons, and don’t really have a problem using $2 double-sided tape to compensate for Motorola’s QA team having the same skill as an inebriated eight-year-old, I think you’ll be much happier with your purchase now than you’d have been a month ago. Otherwise, you might want to wait for the next Android-powered phone on Verizon and see if it works better. It’s certainly unlikely to be worse.

One of the things that really sucks about doing online code reviews is that, in all the systems I know, your code reviews do not integrate with your source control. If the code reviews are versioned at all—and they’re frequently not—then they’re in an entirely different system than your real VCS. For larger reviews, where you’re talking about a major piece of functionality, that means that your source control system will end up lacking the history of how a feature came to be. In other words, the more you use code reviews, the less actual history you have in your VCS.

That’s totally broken. You’re being punished for doing the right thing.

A little over a year a year ago, Tyler came to me and asked me to join him in the Django Dash, a weekend code sprint. Tyler and I had been talking about the code review problem, and had been thinking of writing our own that lacked these issues. Django Dash seemed like the perfect time to try to actually do that.

That opened up a question: how do you actually achieve better code review? If you accept that a huge part of the problem is that the code review history is out-of-stream with your VCS, then it follows that you have to somehow store the in-process code in the VCS.

In most systems, that means using branches. But branches in almost any system suck. Everyone has a horror story about trying to do a merge in Subversion, or CVS, or Perforce—and these are usually not meaningfully large merges; just small feature branches. Trying to use branches for long-running code reviews in these systems simply isn’t viable.

But DVCSes are great at handling this kind of problem. To be distributed, they have to have extremely robust branching and merging systems. Because their systems are so good, it’s very common in DVCSes to do quick experiments and features in their own branch, then merge when complete.

Tyler and I were both big fans of Mercurial (in fact, we convinced all of Fog Creek to switch to Mercurial from Subversion), so using Mercurial as our DVCS base seemed like the best bet. After some discussion of the technical details for making the system work, we got a good night’s sleep, woke up early, threw a nice breakfast on the table, and started coding.

Forty-eight hours later, we had our first prototype. When users wanted to contribute code to a repository, they would fork the repository, push all of their changes to the fork, and then request a review on the fork. Users would see the exact diff of what they would be approving to the repository; no more, no less. Code could not be approved unless it had already merged in the trunk, ensuring that the user who wrote the code had taken care of the merge. When the review was approved, it’d be seamlessly merged into trunk (guaranteed seamlessly, due to the previous rule), with full history.

The design was inflexible and unintuitive, and would have had serious issues in a shipping project, but we achieved what we set out to do:

1. Approving a code review was the same as pushing it
2. Which meant that we could fully separate the concepts of code author and code approver
3. And which meant that the full history of reviewed code was completely preserved

A screenshot of the prototype that would later become Kiln

Even in its nascent form, the tool was already impressive enough, and unique enough, that we won the Django Code Dash.

Tyler and I talked of making our code review tool into a real product, but we were knee-deep in Copilot work, so it had to wait. But we had proved, if only to ourselves, that using a DVCS, even in a centralized model, provided some very unique capabilities that simply were not possible in other systems.

When Joel announced a few months later that FogBugz needed a source control system, we’d be ready.

I’m not entirely sure that this qualifies as a reasonable captcha for mere mortals.

Perhaps on a typographer’s site…

For the past year, an odd thing has happened, if you’ve followed my doings. My work on Fog Creek Copilot seemed to dwindle, I became tight-lipped about what I was working on, and I started getting really excited about an upcoming product release. Also around this time, my knowledge of Mercurial, Python, C#, and ASP.NET MVC all seemed to dramatically increase, even though my free-time code output shrank to nothing. What was going on?

Oh, the usual. I was working on a top-secret brand-new project. And now, it’s released to closed beta.

I’d like to introduce to you Kiln, a brand-new source code hosting and code review tool from Fog Creek. Kiln introduces what I believe is a truly novel take on code reviews that integrates the strengths of Mercurial and FogBugz to provide rejectable code review after commit. We also have some unique takes on Mercurial features, such as the ability to preview what will go into a merge beforehand, really awesome branch management, the most beautiful DAG view I’ve seen in any DVCS product, and lots more.

Over the next few days, I’ll be providing a couple of blog posts detailing how Kiln was developed. In the meantime, go check out Kiln and sign up for the beta. We’re approving new people for the beta on a regular basis, so if you don’t get an invite immediately, don’t worry; we’ll get to you sooner rather than later.

I was delighted to find that Steve Losh has begun making a website called hg tip—a site updated on a regular basis with Mercurial tips for both beginner and expert users. The site’s beautifully designed and a pleasure to read. If you use Mercurial, do yourself a favor and go take a look.

(My favorite tip, incidentally, is definitely the tutorial on making a command called nudge, which allows you to push only the current head by default, rather than all of them. I’ve been using a variant of that, combined with bookmarks, to have Git-style lightweight branching in my Mercurial work when appropriate.)

Finally, a phone whose dev program doesn’t make me want to vomit. Now if only Palm would get the Pre out on Verizon faster, I might actually do so…

“Implementing caching would take thirty hours. Do you have thirty extra hours? No, you don’t. I actually have no idea how long it would take. Maybe it would take five minutes. Do you have five minutes? No. Why? Because I’m lying. It would take much longer than five minutes. That’s the eternal optimism of programmers.”

— Professor Owen Astrachan during 23 Feb 2004 lecture for CPS 108

Accusing open-source software of being a royal pain to use is not a new argument; it’s been said before, by those much more eloquent than I, and even by some who are highly sympathetic to the open-source movement. Why go over it again?

On Hacker News on Monday, I was amused to read some people saying that writing StackOverflow was hilariously easy—and proceeding to back up their claim by promising to clone it over July 4th weekend. Others chimed in, pointing to existing clones as a good starting point.

Let’s assume, for sake of argument, that you decide it’s okay to write your StackOverflow clone in ASP.NET MVC, and that I, after being hypnotized with a pocket watch and a small club to the head, have decided to hand you the StackOverflow source code, page by page, so you can retype it verbatim. We’ll also assume you type like me, at a cool 100 WPM (a smidge over eight characters per second), and unlike me, you make zero mistakes. StackOverflow’s *.cs, *.sql, *.css, *.js, and *.aspx files come to 2.3 MB. So merely typing the source code back into the computer will take you about eighty hours if you make zero mistakes.

Except, of course, you’re not doing that; you’re going to implement StackOverflow from scratch. So even assuming that it took you a mere ten times longer to design, type out, and debug your own implementation than it would take you to copy the real one, that already has you coding for several weeks straight—and I don’t know about you, but I am okay admitting I write new code considerably less than one tenth as fast as I copy existing code.

Well, okay, I hear you relent. So not the whole thing. But I can do most of it.

Okay, so what’s “most”? There’s simply asking and responding to questions—that part’s easy. Well, except you have to implement voting questions and answers up and down, and the questioner should be able to accept a single answer for each question. And you can’t let people upvote or accept their own answers, so you need to block that. And you need to make sure that users don’t upvote or downvote another user too many times in a certain amount of time, to prevent spambots. Probably going to have to implement a spam filter, too, come to think of it, even in the basic design, and you also need to support user icons, and you’re going to have to find a sanitizing HTML library you really trust and that interfaces well with Markdown (provided you do want to reuse that awesome editor StackOverflow has, of course). You’ll also need to purchase, design, or find widgets for all the controls, plus you need at least a basic administration interface so that moderators can moderate, and you’ll need to implement that scaling karma thing so that you give users steadily increasing power to do things as they go.

But if you do all that, you will be done.

Except…except, of course, for the full-text search, especially its appearance in the search-as-you-ask feature, which is kind of indispensable. And user bios, and having comments on answers, and having a main page that shows you important questions but that bubbles down steadily à la reddit. Plus you’ll totally need to implement bounties, and support multiple OpenID logins per user, and send out email notifications for pertinent events, and add a tagging system, and allow administrators to configure badges by a nice GUI. And you’ll need to show users’ karma history, upvotes, and downvotes. And the whole thing has to scale really well, since it could be slashdotted/reddited/StackOverflown at any moment.

But then! Then you’re done!

…right after you implement upgrades, internationalization, karma caps, a CSS design that makes your site not look like ass, AJAX versions of most of the above, and G-d knows what else that’s lurking just beneath the surface that you currently take for granted, but that will come to bite you when you start to do a real clone.

Tell me: which of those features do you feel you can cut and still have a compelling offering? Which ones go under “most” of the site, and which can you punt?

Developers think cloning a site like StackOverflow is easy for the same reason that open-source software remains such a horrible pain in the ass to use. When you put a developer in front of StackOverflow, they don’t really see StackOverflow. What they actually see is this:

create table QUESTION (ID identity primary key,
                       TITLE varchar(255), -- why do I know you thought 255?
                       BODY text,
                       UPVOTES integer not null default 0,
                       DOWNVOTES integer not null default 0,
                       USER integer references USER(ID));
create table RESPONSE (ID identity primary key,
                       BODY text,
                       UPVOTES integer not null default 0,
                       DOWNVOTES integer not null default 0,
                       QUESTION integer references QUESTION(ID))

If you then tell a developer to replicate StackOverflow, what goes into his head are the above two SQL tables and enough HTML to display them without formatting, and that really is completely doable in a weekend. The smarter ones will realize that they need to implement login and logout, and comments, and that the votes need to be tied to a user, but that’s still totally doable in a weekend; it’s just a couple more tables in a SQL back-end, and the HTML to show their contents. Use a framework like Django, and you even get basic users and comments for free.

But that’s not what StackOverflow is about. Regardless of what your feelings may be on StackOverflow in general, most visitors seem to agree that the user experience is smooth, from start to finish. They feel that they’re interacting with a polished product. Even if I didn’t know better, I would guess that very little of what actually makes StackOverflow a continuing success has to do with the database schema—and having had a chance to read through StackOverflow’s source code, I know how little really does. There is a tremendous amount of spit and polish that goes into making a major website highly usable. A developer, asked how hard something will be to clone, simply does not think about the polish, because the polish is incidental to the implementation.

That is why an open-source clone of StackOverflow will fail. Even if someone were to manage to implement most of StackOverflow “to spec,” there are some key areas that would trip them up. Badges, for example, if you’re targeting end-users, either need a GUI to configure rules, or smart developers to determine which badges are generic enough to go on all installs. What will actually happen is that the developers will bitch and moan about how you can’t implement a really comprehensive GUI for something like badges, and then bikeshed any proposals for standard badges so far into the ground that they’ll hit escape velocity coming out the other side. They’ll ultimately come up with the same solution that bug trackers like Roundup use for their workflow: the developers implement a generic mechanism by which anyone, truly anyone at all, who feels totally comfortable working with the system API in Python or PHP or whatever, can easily add their own customizations. And when PHP and Python are so easy to learn and so much more flexible than a GUI could ever be, why bother with anything else?

Likewise, the moderation and administration interfaces can be punted. If you’re an admin, you have access to the SQL server, so you can do anything really genuinely administrative-like that way. Moderators can get by with whatever django-admin and similar systems afford you, since, after all, few users are mods, and mods should understand how the sites work, dammit. And, certainly, none of StackOverflow’s interface failings will be rectified. Even if StackOverflow’s stupid requirement that you have to have and know how to use an OpenID (its worst failing) eventually gets fixed, I’m sure any open-source clones will rabidly follow it—just as GNOME and KDE for years slavishly copied off Windows, instead of trying to fix its most obvious flaws.

Developers may not care about these parts of the application, but end-users do, and take it into consideration when trying to decide what application to use. Much as a good software company wants to minimize its support costs by ensuring that its products are top-notch before shipping, so, too, savvy consumers want to ensure products are good before they purchase them so that they won’t have to call support. Open-source products fail hard here. Proprietary solutions, as a rule, do better.

That’s not to say that open-source doesn’t have its place. This blog runs on Apache, Django, PostgreSQL, and Linux. But let me tell you, configuring that stack is not for the faint of heart. PostgreSQL needs vacuuming configured on older versions, and, as of recent versions of Ubuntu and FreeBSD, still requires the user set up the first database cluster. MS SQL requires neither of those things. Apache…dear heavens, don’t even get me started on trying to explain to a novice user how to get virtual hosting, MovableType, a couple Django apps, and WordPress all running comfortably under a single install. Hell, just trying to explain the forking vs. threading variants of Apache to a technically astute non-developer can be a nightmare. IIS 7 and Apache with OS X Server’s very much closed-source GUI manager make setting up those same stacks vastly simpler. Django’s a great a product, but it’s nothing but infrastructure—exactly the thing that I happen to think open-source does do well, precisely because of the motivations that drive developers to contribute.

The next time you see an application you like, think very long and hard about all the user-oriented details that went into making it a pleasure to use, before decrying how you could trivially reimplement the entire damn thing in a weekend. Nine times out of ten, when you think an application was ridiculously easy to implement, you’re completely missing the user side of the story.

These days, arguing that open-source software is crap seems dumb. How many websites are powered by a combination of MySQL, PHP, and Apache? How many IT applications, written in Eclipse, run on Java, using SWT widgets? How many design studios rely heavily on The GIMP and Inkscape for their everyday photo-retouching and page layout needs?

Er, wait. That last one. Doesn’t quite ring true. In fact, as good as most people seem to insist that Inkscape and The GIMP are, I’ve yet to see a major shop that ran on anything other than Adobe or Quark.

Okay, but, at least I can point to the many offices that run OpenOffice or KOffice! Or that have ditched FileMaker and Access for Kexi! Or that proudly rely on OpenGroupware for their scheduling needs!

…well. Except that I can’t.

I mean, yeah, there are some real businesses, here and there, that actually use those products, and some of them are successful. But, by and large, despite the success of open-source on the backend, open-source end-user applications have failed. In fact, when it comes to end-user applications that people other than open-source developers actually use, you’re pretty much limited to a single application: the web browser. (And even there, if you’re on Safari, then only your engine is open-source—and if you’re on IE or Opera, not even that.) And although I’m sure someone’s gonna say that open-source end-user apps are going to take over any day now, they’ve been claiming that since the height of the dot-com bubble, ten years ago. I wait with bated breath.

Why does open-source fail to reach critical mass anywhere but the server closet?

Easy: because open-source software is, incontrovertibly, a total usability clusterfuck.

Programmers are superb optimizers. We’ve been accused of being lazy in the best way possible: we try to ensure we do not solve problems that do not need to be solved, and that we solve those that we must deal with completely, so that they never bother us again.

On open-source projects, where anyone can quickly nail that one little bug that was ticking them off, that means that your software is gonna be lean (why implement what you won’t use?) and operate to spec (you don’t wanna keep dealing with that one annoying bug every day). So far, so good.

But what about using software? You only gotta learn software once—and, for those actually contributing on an open-source project, you probably learned from the bottom up, so that’s how you view the thing. Interface elements that expose quirks of the underlying implementation seem totally natural; what a user perceives as a bug strikes a developer as little more than the reflection of the underlying system.

Developers could fix this problem. They just completely lack motivation. Being “lazy” at this point means leaving the software as-is. If users find the software frustrating and unintuitive, or can’t get the thing installed, they should spend the time to learn the underlying, beautiful implementation, at which point they will discover a world of awesome and inspiring flexibility far greater than what closed-source offerings could possibly provide. And, until then, go bug the mailing list so that we can all call you an idiot.

What the developer-as-lazy argument misses is that companies, too, are lazy—and, if competently run, lazy in a good way. Employee time is money, and therefore a smart company will attempt to reduce how much time its employees must spend on a problem. If companies only employed developers, the end result would be the same as the open-source model. But they don’t. Companies also have support staff, and the amount of time support staff must spend fixing problems is directly proportional to how well-written and intuitive the software is.

Note that second part: any time that a user gets confused, and has to phone support to get through a problem, it costs the company money. The company is highly motivated to produce easy-to-understand and easy-to-use software to keep its support costs down. After all, at the end of the day, a user does not care how robust, or how elegant, or even how beautiful, the implementation of their software is; all they care about is whether they can use your video software to make glitter fly out their plastered boss’s butt in the Christmas party video.

So, if faced between spending time on the elegance of the implementation, or the intuitiveness of the interface, companies will optimize for intuition; open-source projects, for elegance of implementation.

In the general case, the open-source emphasis is wrong—at least, if you want your software to actually get used. All of the oddball exceptions in open-source usability—Firefox, Firefox 2, Firefox 3, and I guess GNOME, if you make twist my arm—have massive corporations backing them up, thinking in both development and support costs, and spending the time to make sure that users have a positive first-time experience with the software. Without exception, these products are only open-source because they, as a product, don’t actually confer much value to the parent company. Mozilla ultimately cares far less about whether you actually use Firefox than whether your Google queries list Mozilla as the referrer; Sun, at least in its drunk camel of a business plan, cared more whether you were running on Sun hardware than whether your desktop happened to run GNOME over KDE. Open-source, without corporate guidance, cares more about making sure you can tweak your cluster size to match the optimal expression of K-trees on BlenderFS on inverted Xeon cache vertibrates without affecting the MIPS port, than about ensuring that a new user has the faintest idea how to do anything with that package he just downloaded.

Maybe, someday, human altruism will make possible the Grand Dream of Open-Source, where projects are open-source, and well thought-out, and easy-to-use, and easy-to-install, and highly efficient, and bug free. Until then, open-source software is going to run great, but be painful to use, and closed-source software will be easy, but less efficient. Pick which you want for your own purposes. Just don’t forget to take a look around at how much Apple hardware you see these days to figure out which users actually value.

In its notes on Snow Leopard, Apple claims:

64-bit computing shatters [the 4GB memory] barrier by enabling applications to address a theoretical 16 billion gigabytes of memory, or 16 exabytes. It also enables computers to process twice the number of instructions per clock cycle, which can dramatically speed up numeric calculations and other tasks.

Wrong, Apple. Ridiculously wrong. It might enable you to process twice as much data per clock cycle, depending on what you’re doing and how the chip is architected, but even that’s best-case. This reads like the fanboy arguments of PowerPC versus x86 from the late 90s—but with less intelligence.

Hopefully Apple will fix its wording before they ship Snow Leopard, or I’m sure some lawyer somewhere will have a wonderful time filing a lawsuit for false marketing.

To make it easier to follow the goings-on of Star Trek, Google Calendar now helpfully lets you know which Stardate things occur on:

I eagerly look forward to not using this feature as soon as possible.

I’m thrilled to say that, after two years of heavy development, my colleagues have released FogBugz 7 Beta. If you’re interested in joining, head on over to the signup page to request to have your license or FogBugz On Demand account upgraded. The existing beta already adds a heavily revamped interface and tons of new features, but I have a hunch that there may be even bigger surprises in store. Keep an eye on the FogBugz team in the weeks to come.

When I first saw Palm demo the Palm Pre, I was very excited: for the first time, I saw a phone that looked as if it would be a genuine competitor to the Apple iPhone. Palm has clearly put lot of thought put into its design: the interface is intuitive and fluid, basically invents a reasonable way to do multitasking on a portable device, and still seems to adhere closely to Palm’s historical emphasis on simple, clean UIs—something that my nine-year-old Handspring still does better than my BlackBerry. Yet many people, including me, were somewhat dubious that the Pre in its present form would be able to come to market. Much of its appeal comes from the fact that it clearly owes much to the iPhone—which is heavily patent protected. Apple would stand to gain a lot by attempting to prevent the Pre from ever reaching consumers.

Thankfully, Palm seems to have the upper-hand. Engadget has a wonderful analysis of the iPhone’s actual IP—which, it turns out, is fairly limited—and also points out that the iPhone itself clearly violates quite a few Palm patents. After reading the article, I suspect no lawsuits will be filed; Palm would file a rather strong counter-suit, and the entire thing would just get settled out-of-court with a cross-licensing agreement.

The only remaining question is whether the Pre will actually make it worth using Sprint, arguably the weakest major cell network. Sadly, that’s largely out of Palm’s hands.

Not outsourcing to India; outsourcing to third parties. The Wall Street Journal details how Sony’s R&D budget paid for the CPU in Microsoft’s Xbox360, and how that ended up being part of the reason for the 360’s success against the PS3. Joel’s defense of NIH syndrome for core functionality in your product makes more and more sense the longer I spend in the industry.

I normally avoid reposting news I find on other news aggregators, but sometimes I come across an item sufficiently singular and unique that I feel I have no choice. In this case, Adam Savage of MythBusters recounts his attempt to sculpt a perfect recreation of the Maltese Falcon as a way to explore the nature of obsession. The talk is at once highly entertaining and deeply moving. In a way I never fully grokked when watching MythBusters, Adam is a true geek.

Several weeks ago, I ran across a small OS X tool called The Unarchiver—a free replacement for the built-in BOMArchiverHelper.app utility that Mac OS X uses to extract zip archives. The Unarchiver goes beyond BOMArchiveHelper, handling more file formats (including 7-zip and StuffIt), including better internationalization support, and having a more Finder-like interface (multiple simultaneous extractions are in a single window). I’ve been very happy with it. If you run OS X and deal with archives often, take a look.

I’ve long preferred Pownce to Twitter. It allows longer messages, has better uptime, has a better API, allows you to send links, photos, and files in addition to text…basically, it was just better in every way. The Copilot team made heavy use of Pownce during our Hair on Fire sprint, and continued to use it until we adopted Laconica internally.

So I was deeply saddened to learn that, in a mere two weeks, Pownce will be no more. I guess Twitter’s where I’ll be now, whether I like it or not.

Pownce, you’ll be missed.

Just remember: nothing says traditional Thanksgiving like the Honeywell H316 Pedestal “Kitchen Computer”.

I’ve been a big fan of Dropbox, a completely seamless remote storage solution, since they first had their closed beta. They provide two gigabytes of storage for free, have native clients on Mac, Windows, and Linux (including 64-bit!) that Just Work™, and have a ridiculously clean website that makes seeing a history of changes, and even restoring deleted files, wonderfully simple.

Recently, Dropbox has been getting some love for how awesome it is, and as I was poking through Hacker News comments on the service, I discovered yet another reason Dropbox rocks: they have an iPhone interface that lets browse your photos and documents, and see any recent changes to your folders. Basically, anywhere you have an iPhone, you have access to your Dropbox.

It’s little things like this that make me think I really should have gotten an iPhone instead of a Blackberry Pearl.

A company named oblong has implemented the Minority Report computer interface in real life:

You can read more on their corporate blog.

Would we recommend the Storm? If you’re locked into a contract with Verizon, want a touchscreen phone, and are willing to put up with an OS that moves like a tranquilized yak, then yes the Storm is for you.

Not exactly the rousing endorsement that Verizon and RIM were hoping for, I think, but consistent with what I expected. Verizon is utterly terrified of giving up any control over its cellular network. That may seem like a good idea in the short term, but with T-Mobile and AT&T making non-trivial concessions to consumers, both in features and flexibility, it can only work for so long.

I recently upgraded my laptop to Ubuntu 8.10, and amazingly, Ubuntu just went from one of the worst font systems I’ve seen on any system to the best I’ve seen on any system. They seem to have taken Microsoft’s ClearType technology and improved upon it surprisingly well.

I’m proud to announce that we’ve just shipped Fog Creek Copilot OneClick. You can read about the details at the new Fog Creek Copilot blog, Air Traffic

I love OS X, and in general find Apple’s software on the platform extremely high-quality, but when it comes to running Apple’s software on Windows…well, it’s an entirely different ballgame. Safari uses its own widgets, windows, and font rendering, which makes it look utterly out of place on my desktop. QuickTime for years simply looked bizarre, using a window to hold just the menu bar, and then other windows to hold the movies. iTunes uses Windows-native font rendering and some native widgets, but mostly tries to foist OS X paradigms on Windows users for no particular reason. And all of these applications on Windows use tremendous amounts of RAM and CPU.

Yet even I have to admit that iTunes 8’s stability issues take Apple’s past poor performance to an entirely new level. Installing Safari with an iTunes update some months ago was bad enough, but installing a poorly vetted driver and a fleet of MobileMe services that cause Windows to lock up hard is inexcusable. For all people like to rant against Windows, the OS, at its core, honestly is very stable; it really is almost always third-party drivers that cause issues. In this case, that third party vendor is Apple.

I don’t think Apple is writing crappy Windows software on purpose; I just think that their Windows team is grossly incompetent. It’s past due that Apple either starts writing non-sucky Windows software, or quits trying.

Google has begun digitizing old newspapers, making certain old Onion stories a bit less funny.

I hope it goes without saying that I love technology, but…at the same time, there’s something I used to find infinitely more gratifying about having to use card catalogs, paper indexes, and microfiche. That romantic nostalgia makes me keep my diaries in Moleskines and sketchbooks, causes me to allow piles of books to keep refuge in my apartment, strikes fear into my heart when I see the disturbingly named Kindle, and explains why I may own cutting-edge computers, but can’t give up my fountain pens. As much as technology buys you speed, it costs you personality, until the only concrete objects you interact with do little more than reify equations and thoughts into some transient form you can ingest and vomit out at a later date. When you can acquire information with no effort, its value disintegrates. Sometimes, I worry that this yielding to an ephemeral reality, not a loss of privacy, is the price we pay by having Google around.

You have successfully discovered Morphic.

Can you discover Smalltalk too, as long as you’re at it?

In other news, Honda takes a cue from Microsoft and makes a unique car that totally doesn’t borrow any ideas from any of its competitors. Keep on bringing that innovation, Honda.

I think T-Rex may have a point, for once…

Braid’s been receiving accolades for its amazing gameplay and complex storyline. Over at Gamasutra, I stumbled upon a great article detailing the evolution of Braid’s artwork. Especially after having beaten the game, I found it fascinating to see how the art had evolved, and had helped to give the game its unique feel.

Moving us one step closer to Blade Runner, I’m pleased to bring you a fully robotic water snake. Although the snake at first appears to be little more than a toy, its movements are preternaturally organic. I have a very easy time believing that I’m looking at some type of exotic life form—and I find that simultaneously amazing and frightening.

Philip K. Dick would be proud.

Although I’m not especially worried by higher fuel prices—in my opinion, they’ll help accelerate a much-needed movement to electric vehicles, which in turn will force us to use more nuclear and solar power—one of the things I’ve routinely wondered is how planes will deal with the problem. To me, the only solution seemed to be to use hydrogen (rather explosive) or increasingly expensive gas propellants. It turns out, though, that small planes can be powered by batteries—and someone has already made a battery-powered propeller plane. It’s quiet, clean, and can fly for 90 to 120 minutes—enough for a pleasure cruise. And the best part? It costs a mere 60¢ to refuel the craft.

Count me in.

Given I know a few people myself who don’t own cellphones, I found defective yeti’s list of how people’s reaction to the discovery he does not have a cell phone has changed over the course of the last decade spot-on and hilarious.

And it turns out that the Toyota Prius isn’t necessarily that great for the environment after all. (This should not come as a surprise if you’ve been keeping up on the research into renewable energy.) Listen closely to the end of the segment, though—the point isn’t that the Prius cannot be more efficient than the M3, but rather that the driver has to do his part to drive more conservatively, too—something that I’ve argued, and been keenly aware of, since I started driving.

Bell Canada is currently engaged in a lovely kerfuffle with the CRTC (Canada’s rough equivalent of the FTC) for throttling BitTorrent traffic. The CRTC recently ordered Bell Canada to release its bandwidth numbers, and Bell Canada, after some protestations, complied. The little teensy problem with their data, as Ars Technica points out, is that the numbers indicate that any problems Bell Canada is experiencing have nothing whatsoever to do with BitTorrent, and can be trivially and cheaply fixed.

I am shocked—shocked!—to find that gambling is going on in here!

This has got to be one of the dumbest concepts for a cell phone I’ve seen in my life. I’m kind of amazed it wasn’t killed in preproduction.

It may not enable time travel, but the flux capacitor, in a literal sense, is here. Called a memristor, the device provides similar functionality to a transistor, but at vastly higher efficiencies, an should allow for much smaller, more efficient computers in the future.

The Freakonomics Blog has a fasciating report on the horrible accuracy of TV weather stations. Although I don’t find the results remotely surprising, the data reflect such a profound lack of insight that I’m forced to reevaluate whether watching the weather is worth my time at all. For most people, going outside, looking at the sky, and paying attention to changes in humidity seems as if it would yield more accurate results.

Sun has just announced that they will begin close-sourcing MySQL. For years, I’ve avoided MySQL due to a mixture of paranoia (I’ve had extremely bad experiences with MyISAM-backed data stores) and disdain for their shoddy standards compliance (which has bitten me before in nontrivial ways). Now I can also avoid them for not being open-source.

My standardization on PostgreSQL for this website feels more rational by the minute.

Update: The originally linked article wasn’t quite correct. MySQL AB’s CEO explains that they will not be making the MySQL core closed-source; merely new, enterprise-specific features. You can read his whole statement for more information. This position is more reasonable, and is similar to the relationship ElephantDB has with PostgreSQL.

Tim Bray makes the same argument I’ve been making for months on why ISO-certified OOXML won’t actually make a lick of difference. At least the ISO has successfully proved how corruptible they are for all geeks to see, so I suppose the approval process wasn’t totally useless.

Yahoo just announced that they’re buying a company that makes a competitor to Google Analytics and releasing it for free. This is the first rational thing I’ve seen coming from Yahoo in quite some time. Although I’m not a big fan of the “buy product, release for free” business strategy, a web traffic analyzer is a perfect loss-leader for Yahoo, because it should boost their ailing ad program. The value of Google Analytics for Google is that it helps me maximize my AdWords revenue. That means more immediate revenue for Google and increases the chance I’ll use them again in the future. Now that Yahoo’s own ad program will have a similar tool, they should be better positioned to get and retain business from web publishers. Whether that actually ends up being the case depends on a large number of things—the quality of the purchased product, how well integrated it will be with Yahoo Publisher—but at least it’s a step in a very good direction for Yahoo.

As you probably noticed, WordPress got swapped out for MovableType last night. The good news is that I’m rapidly falling in love with the new system and expect to have this site back with sane templates by the end of next week. The bad news is that all the user accounts were lost. All comments have been preserved, mind you, but if you previously had an account, you’ll have to reregister.

On the bright side, I’ve opted to experimentally enable OpenID support. I’ll be monitoring it closely—I have a bad feeling about OpenID making spammers’ jobs a bit too easy—but for the time being, please feel free to comment using your OpenIDs. With luck, this blog can be one of the first to begin eliminating site-specific accounts.

The Coding Monkeys have released Port Map, an application to make accessing computers behind firewalls NATs easier. Unlike Copilot, which tries to work around obstinate routers, Port Map focuses on providing an easy and consistent interface for reconfiguring them. It’s hardly perfect for everything—notably, you have to have permissions to reconfigure the router causing you difficulty, making it unsuitable for corporate environments—but I can see it being quite handy if you’re just trying to ensure you can reach your home computer from work to grab a couple of MP3s.

I upgraded bitquabit to Ubuntu today. I learned a few valuable lessons:

1. Untested backup scripts don’t count. This one I knew, but I didn’t fully process that “untested” really means “untested recently.” In particular, my backup script was backing up a database called wordpress. Unfortunately, I moved all the blogs hosted by bitquabit to a database called wp last fall. Result? The backups, though minutes old, were effectively from last October. I was lucky here: I happened to have a day-old WXR file, and a friend sent me the one missing post that was still on his screen in Google Reader. Another blog hosted on bitquabit was not nearly so lucky; it will basically have to start from scratch.
2. Backups tested on systems substantially different from the deployed system are useless. I love Citadel; it’s an awesome groupware client. And my backups and restores worked quite well—when both ends were running Debian 4. Unfortunately, I just moved bitquabit to Ubuntu 7.10. You can guess the rest.
3. Have multiple backup strategies. Although the Citadel “migration” didn’t exactly go as planned, I was saved there by paranoia: I sync my IMAP mailboxes to my local machine in mbox format. I was back up and running about a minute after the server came back up. No data loss there.
4. Don’t migrate at odd hours if you don’t have to. Realistically, my last-minute checks would at least have caught the fact that the blog database was borked if my brain had been more awake. The “most recent” post was, after all, more than six months old, and had encoding errors that I’ve fixed before. Then again, I didn’t catch this in the dry run on Sunday, either, so I’m not quite sure how much I can meaningfully excuse my idiocy through fatigue.

On the bright side, everything, minus a blog, seems to be fully up and operational again. There are a couple of quirks—with the blog hell, I opted to do an impromptu migration to MovableType—but I’m confident I can get things straightened out again. Meanwhile, all I can do is live and learn and hope this is the last time I have a lossy server migration.

According to Ars Technica, parents would rather that their kids’ video games feature decapitations than sex. I don’t really have any commentary to add; just read the whole article.

One of the cool new features of Mac OS X Leopard is Time Machine, a really simple backup solution for Mac OS X that not only transparently backs up your data, but also does so with an amazingly ugly GUI that lets you quickly jump back to the way that your documents were at any given point in the past. Unfortunately, Time Machine doesn’t run on my Linux boxes, so I’m forced to come up with an alternative.

The good news is that getting a 90% solution is ridiculously easy. On the back-end, all that Time Machine does is create a collection of hard links from one backup set to another. Here’s the bare minimum of a shell script that will back up the last three editions of your home folder to an external drive:

#!/bin/sh
SOURCE=/home/benjamin
DEST=/mnt/tardis
rm -fvr "${DEST}.3"
mv "${DEST}.2" "${DEST}.3"
mv "${DEST}.1" "${DEST}.2"
mv "${DEST}.0" "${DEST}.1"
rsync -av --exclude-from='backup-excludes' --link-dest="${DEST}.1" "${SOURCE}/" "${DEST}.0/"

You’ll obviously need to modify the SOURCE and DEST variables to be something appropriate for your computer. You’ll also need to create a file in your home directory called backup-excludes that’ll look something like this:

Documents/Code/3rd-party
Documents/Code/Builds
.emacs-backups

Add and modify the glob patterns so that it contains a list of things that should be skipped. Now, just make sure your external drive is plugged in, run the script, and presto! Instant backup. Quick and dirty, but gets the job done.

Hacking isn’t limited to pro-Tibetan groups; the Pentagon notes that cyberattacks against US defense infrastructure has greatly increased in the last few years. Given the sad state of computer security and the increased use of consumer components by the military, I strongly suspect that the average American would be petrified to learn how many national secrets we’ve failed to protect. For the time being, at least ignorance truly is bliss.

I’ve been against software patents for a long time now, but when I read about stories such as satellites being turned into space garbage because the only way to fix the orbit is patented, I’m forced to question the wisdom of patents in general. I love the idea of patents; I’m just dubious that the current implementation actually works. More often than not, I see patents used not to protect a novel invention, but as a legal stick to bludgeon small competitors. That runs completely against the original intention of patents, and leaves us as a country poorer.

If Google has any sense at all, they’ll sit back, order some popcorn, and watch their competition commit suicide.

Couldn’t have said it better myself. My running hypothesis had been that peyote was secretly legal in Redmond. I evidently need to add Sunnyvale and Loudoun County to that list.

Sadly, there’s no such thing as NetNewsWire for Windows, but today I downloaded FeedDemon, which is made by the same company and also offers synchronization with NewsGator, and was extremely pleasantly surprised. If you, like me, have a Mac at home and a PC at work, it’s definitely worth a look.

As I indicated curtly in my previous post, I’m a huge proponent of nuclear power.

Though there continues to be substantial political debate whether global warming exists—largely because responding to it would be economically damaging—the overwhelming consensus of the scientific community is, and has been for some time, that global warming exists and is man-made. (See the IPCC statement, and a discussion of its significance in Nature—one of the top several scientific journals in the world, and definitely representative of the scientific community—for the most recent affirmations of that claim.) Even for those who refuse to believe in global warming—whether because they believe that the overwhelming majority of scientists and their communities are corrupt, or that scientists are incompetent, or that G-d will prevent climate change—few would argue that reducing pollution, if economically viable, is a worthwhile goal.

Nuclear energy provides a cheap, reliable, highly efficient way of generating electricity right now. Combined with a movement away from fossil fuels, nuclear power would offer cleaner air and cheaper power.

Though many argue nuclear power is unsafe, I believe their fears are largely unfounded. Chernobyl and Three Mile Island are the only two nuclear-power-related accidents we’ve had over the past fifty years, and only Chernobyl had radiation leakage. Given that 443 reactors have been built and are currently operating (not even counting secret and naval reactors) according to the IAEA, fears of nuclear-power apocalypse seem overblown. Nuclear plants, meanwhile, generate no air-based pollutants, in stark contrast to coal plants, which generate relatively high levels of toxic pollutants. Given the choice, I would much rather live close to a nuclear power plant than a coal power plant.

I have far more sympathy with those who argue that countries with nuclear reactors have access to material for nuclear bombs. Sadly, far too many countries today would indeed jump at the chance to create and use nuclear weapons. Though this criticism doesn’t apply to the thirty-two countries who already have nuclear power, and therefore should not be an argument about increasing the use of nuclear power in the United States, I do think that proliferation is a viable concern with spreading the use of nuclear power in the world at large.

Thankfully, we may soon have the best of both worlds: thorium reactors may soon become a reality.

Thorium reactors, unlike uranium reactors, do not produce plutonium (and in fact, will happily, cleanly destroy plutonium as part of its reaction process), and as a result, their waste products remain radioactive for only 500 years. They’re also safer: the thorium fuel cycle is sub-critical, meaning that, in the absence of human intervention, it will burn out quietly, rendering Chernobyls and Three-Mile-Islands are impossible. Thorium is also far more plentiful than uranium, being up to 550 times more plentiful in the Earth’s crust, meaning that such a reactor would be even cheaper to operate. On paper, thorium should be perfect.

Yet thorium has a major flaw: because thorium reactors are sub-critical, they require small amounts of uranium and plutonium to keep the reaction alive, which results in a slight catch-22. Even though such a hybrid plant would be far safer than a pure uranium- or plutonium-based reactor, it accomplishes nothing to assuage anti-proliferation fears.

The good news is that this will change in the very near future. Cosmos Magazine has a great article on two new ways of powering thorium reactors—the second requiring no uranium or plutonium whatsoever, instead using a particle accelerator powered by the reactor itself to keep the reaction running. Such a reactor would offer cheap, clean, powerful fuel to power our world well into the future with minimal environmental or social repercussions.

I fully anticipate a long wait before thorium reactors make an appearance in the United States, but unless fusion power finally proves viable—something I don’t think even ITER will help achieve in the near future—thorium promises to be one of the best options for our future energy needs.

I’m not quite sure how I feel about the following dialog I got in Interface Builder 3 when trying to load Copilot Mac Helper’s NIB file:

I mean, I guess it’s nice to have a choice, but…is this really the best UI Apple could come up with?

Devin pointed me to an incredible video. Apparently, German computer scientists have figured out how to split a musical recording into its component notes, allowing you to manipulate a digital recording of a piece as easily as if it were just a MIDI recording track. This means that you could generate a cappella versions of your favorite song, or make an artist sing in harmony with herself, or simply fix a one-note recording error, all without having access to the original master tracks or doing any additional recording. You can see the (admittedly corny) video for a great overview of what they’ve accomplished and some great examples of what this technology makes possible.

Electronista has a good video of Android running on a reference platform. The video gives a nice feeling for what a touchscreen-focused Android phone would be like. The result’s about what you’d expect: not nearly as smooth an interface as the iPhone, but significantly better than many existing smartphones.

Personally, although I look forward to Android’s release and am extremely interested in what applications its fully open architecture will make possible, I’m inclined to wait for the second-generation devices. For actually getting things done on a phone, I have a strong suspicion that the BlackBerry Pearl and iPhone are going to remain the best two options for most people.

Modern forensics experts have assembled a picture of what Bach looked like.

Today, I was over at a friend’s house and got sidetracked talking about music we liked. I mentioned that I’d recently discovered Jonathan Coulton and really liked his music and played her a few songs of his. She liked them and asked whether she could have a copy. Since his songs are all licensed under the Creative Commons, that was no problem.

Unfortunately, the only copy of the songs that I had were on my iPod. As everyone knows by now, Apple makes it very difficult to copy songs off an iPod due to piracy concerns. On a Mac, at least, it’s easy to see the song files—they’re just under /Volumes/[iPod name]—but because their names are completely randomized, it can be hard to find any single song. Since ID3 tags are preserved, you can traverse the entire file hierarchy, scan the tags, and copy those files that match, but that solution doesn’t work either if all your files aren’t MP3s or if you have a prohibitively large number of files. In this case, I needed to extract about 40 songs out of several thousand, so I wanted something that would work faster.

The good news is that this problem is blissfully easy to solve with a little shell script and the magical tool lsof. Simply get iTunes displaying all the songs you want to copy, either with a search or a playlist, hit “Select All,” and then execute the following script:

#!/bin/sh

COUNT=`osascript <<EOF
set num to 0
tell application "iTunes"
    repeat with song in selection
        set num to num + 1
    end repeat
end tell
return num
EOF`

mkdir ~/copies
for ((n=0;n&lt;$COUNT;n+=1)); do
    lsof | grep iTunes | grep mp3 | awk '{print $9}' \
         | xargs -J % cp % ~/copies
    osascript -e 'tell application "iTunes" to next track'
 done

Most of the script is self-explanatory, but some notes on the more interesting parts:

1. AppleScript supports a count element on most collections, but for whatever reason, count of selection is always 0. I still have a hunch this is a PEBKAC issue on my end, but I don’t mind kluging around it
2. Bash actually supports C-style looping, which for some reason seems to catch a lot of people by surprise.
3. lsof is an incredibly useful command. It stands for “list open files,” and lists all files (and sockets) that are open on your machine, along with which program opened them. We filter down to MP3 files opened by iTunes.
4. awk has largely been replaced by more powerful scripting languages, yet it still has its uses—here, serving as a quicker and more powerful cut to extract the ninth column of whitespace-delimited text, which happens to be the full path to the open file.
5. BSD’s xargs command allows you to specify an explicit marker for inserting text with the -J option, which we use here so that we can manipulate what’s being copied while holding the destination constant.

So, basically, we just make iTunes play each song long enough to find where on the iPod it’s located, copy it to a safe location, and then tell iTunes to advance to the next track. It’s not especially fast, but it’s far faster than walking the entire iPod looking for the relevant songs. On my system, it took iTunes about twenty seconds to run for forty songs, which is much better than the alternative.

There are relatively few times I honestly find myself wanting to copy songs from my iPod, but the next time you find yourself in a similar situation, you can use the script above to get your copies done quickly.

NetNewsWire, an outstanding RSS reader for Mac OS X, is now completely free. If you own a Mac and haven’t taken a look, now would be a great time. Combined with NewsGator (also now free), you’ll have a great RSS reader for your iPhone or iPod touch, too.

Though many found Apple’s keynote yesterday underwhelming, and certainly little in the keynote was revolutionary, I’m quite excited about some of their announcements. The MacBook Air, despite the whiny criticism it seems to inspire, looks as if it will be an absolutely superb laptop. (I was originally going to write an article about why the criticism thus far against the Air is ridiculous, but Wil Shipley beat me to the punch with his usual mix of whit and rancor, so just read his rant instead.) Although I don’t personally have a use for Time Capsule, I know a large number of less technically minded friends and family that would reap huge benefits from its transparent backups, and I have to confess that I’m actually excited about the new iTunes rental service.

Yet I find myself truly angered by the iPod touch upgrade.

Early iPod touches lacked some of the applications that the iPhone has, including key productivity applications such as notes, maps, and mail. Apple announced that all new iPod touches shipping will include these applications—but that existing users need to pay $20 for them.

Some apologists have claimed that Apple was forced into charging existing iPod touch owners $20 for for accounting purposes. Although that explanation is cute, it’s wrong. No other manufacturer even pretends this problem exists. Opera happily issues free upgrades for their browser on my Wii. Nokia just released a massive free upgrade to my N800 that has literally changed how I use their product. Even Microsoft, whom everyone always likes to accuse of being Satan incarnate and interested in nothing but money, has no problems making free updates that include a raft of new features. The Windows XP service packs included support for USB 2 and .NET applications, massive upgrades to the built-in firewall, a brand-new malware removal tool, greatly enhanced graphics acceleration capabilities, upgrades to Windows Media Player so great that the current version of the application is literally unrecognizable as what originally shipped with XP, and even a tremendous update to Internet Explorer that turned it from a has-been into a decent competitor for Firefox. Microsoft provided these updates for free to all Windows XP users—even ones who bought XP back in 2001 when it was initially released, seven years ago. Apple can’t seem to provide updates to users who purchased their product four months ago.

Even Apple itself has no problem providing new features for free—when they feel like it. As recently as yesterday, the Apple TV shipped with a massive software update that allows the device to contact iTunes directly, frees it from needing to be tethered to a PC, and even the ability to browse photos on Flickr. Apple doesn’t even have a problem updating iPods, as long as the features help them move iTunes purchases. If Apple really cared about their accounting as they claim to, then they ought to be charging users a dollar to add the ability to watch iTunes rentals on their iPods as well. After all, the iPods as-shipped lack this capability, so by any reasonable standard, it qualifies as a new features. By Apple’s own accounting rules, they ought to be charging for it.

Whether Apple claims to be charging users for accounting reasons or no, the actual reason is quite simple. Apple discovered quite awhile ago that their early adopters are willing to pay more for their products. It’s why the iPhone dropped $200 in price after just a month or two, it’s why they charged $2 to active 802.11n wireless on early machines that had it, and it’s why Apple’s charging $20 to early adopters of the iPod touch now. It’s a very deliberate decision on their part to fleece their early users.

Perhaps this will work for them for the time being; I don’t know. What I do know is that they are walking a very dangerous road with these decisions. Early adopters are normally willing to pay a bit more because Apple products are a status symbol. Those with new Apple products are sending a message. By repeatedly screwing their early adopters, Apple is rapidly turning that message from “I am a hip part of the Apple revolution” to “I happily take it from Apple because I’m a corporate whore.” Combine that with the fact that many products only become popular and chic due to those early adopters—the iPod serving as the quintessential example—and Apple could rapidly find their nickel-and-dime business tactics leave them with nickel-and-dime revenues.

I think that Gizmodo performed one of the cruelist, most hilarious hacks I’ve ever seen: they took a TV-B-Gone from MAKE and used it to switch off whole banks of televisions at CES. On the one hand, I feel bad for all the technicians who had to try (and likely failed) to figure out why all the TVs all over the floor were dying, but on the other hand…

…well, just watch for yourself.

Yesterday, I went to a website that used MathML to display a few formulae. Because Firefox supports MathML, I figured everything would display just fine. Unfortunately, Firefox notified me that I had to download some free fonts to display the equations. Here’s the dialog it displayed:

I don’t mind having to install fonts, but this dialog is so poorly constructed that I ended up laughing:

1. The link they provide isn’t clickable. I’m in a frakking web browser, and they’re not going to let me click the link that lets me download the fonts? Seriously?
2. Not only can you not click the link; you can’t even select it (at least on OS X). You have to open another browser and type it in by hand.
3. Firefox tells me which fonts it needs. If I actually navigate to that page, I can’t download those fonts individually; I can only grab all the fonts at once. I am completely okay with having to download all the fonts at once—I even think it’s the right solution—but then, why bother telling me which individual fonts I’m missing if I have to download them all together anyway?
4. For that matter, why is Firefox making me download the fonts? Firefox already knows where they can be fetched, and it knows what platform I’m on, and it has a download manager. It could automate the entire process for me quite easily.

The dialog they present should look like this:

I’ve made a number of improvements:

1. Useless information has been removed. Most users have no idea what MathML is, and even technically proficient ones don’t care exactly which fonts Firefox needs to render it. All users care about is whether the next page they view is legible. The new design clearly explains the problem and the consequences for not fixing it, without going into pointless details.
2. The install process has been automated. No more “go here for more information.” Just install or don’t, right now.
3. For users who actually want more information, a Mac OS X help button has been added that can give them all the nitty-gritty details.

The new dialog is clearer, involves less work from the user, and follows the Mac OS X HIG.

Mac developers in general seem to have a better feel for this kind of thing. That’s actually part of what keeps me wedded to the platform: Macs, for as long as I can remember, seem to attract a higher caliber of user interfaces, even (sometimes especially) in third-party apps. Windows is horribly erratic, and Linux, with its multitude of conflicting interface guidelines, frequently ends up being a Frankenstein of pain.

Chicken chicken chicken chicken, chicken chicken chicken chicken. (Chicken chicken chicken—chicken chicken chicken—chicken chicken.) Chicken chicken chicken chicken chicken chicken! Chicken chicken chicken chicken chicken.

I normally don’t link articles I see on reddit on the theory that you’ll already have seen them, but this one was too good to pass up. New Artisans has a superb article on iptable tricks to defend against common attacks, and even provides some hands-on examples of what they’re defending against. If you administer your own server, I strongly recommend taking a look.

(Note: try the “attacks” they show only within your own LAN. A competent ISP may notice what you’re doing and shut you down if you try some of the simpler things they talk about. Besides, it’s just plain not nice.)

Unix needs an undo command.

This morning, my roommate and I hauled out some of our “big iron” (a languishing Pentium 4 box) to use as a photo server. Because we had initially planned to use that box to host bitquabit.com and its sister sites—a plan since scrapped—it had a full clone of all the data on my Linode hub. Before my roommate got going, then, I thought I’d quickly clean the box and return it to a neutral state. First stop, hose the duplicates of the websites I host. Fire up SSH, sidle into /var/www, double-check with pwd I’m where I think I am, fire off an rm -rf *, and check that the directory’s clean. It was, so I decided to write a message to my roommate. Since I couldn’t remember what his Unix login was, and I knew he was logged on, I ran the w command.

benjamin pts/0    fcfwbeac.fogcree Fri18    2days  1.20s  0.08s citadel
benjamin pts/1    192-168-40-51.c3 21:26    1.00s  0.06s  0.01s w

I don’t remember even installing Citadel on this machine, I thought. And why am I the only one on the server? As very, very dark thoughts started to wander out of my amygdala and set up shop in my frontal lobes, I decided to check that w wasn’t malfunctioning.

benjamin@bitquabit ~> w
9:43:23 up 18 days,  4:29,  2 users,  load average: 0.00, 0.04, 0.01
USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT
benjamin pts/0    fcfwbeac.fogcree Fri18    2days  1.20s  0.08s citadel
benjamin pts/1    192-168-40-51.c3 21:26    1.00s  0.21s  0.00s w
benjamin@bitquabit ~>

I wonder why my roommate’s not showing up, I wondered to myself. Maybe if I run it one more time…

And then it hit me. benjamin@bitquabit. benjamin@bitquabit. Oh frak, I hosed my production server!

But no, wait. I’d set up the spare box to be a bitquabit clone; it makes sense it’d think it was bitquabit. So I dropped out of ssh and read back .ssh/config:

benjamin@bitquabit ~> exit

Good bye
Connection to bitquabit.com closed.
Mungus:~ benjamin$ cat .ssh/config
Host vera
HostName 192.168.1.10

Host bqb
HostName bitquabit.com
Mungus:~ benjamin$

…frak. Sure enough, I hosed all the websites on my production server. Oh, what jolly day.

There were two good things: first, although backup on my server wasn’t automated, I had written a backup script, and it had everything except four images from a very recent article and my hit tracking system, Mint. Second, I got motivated to actually automate my backup system.

Now, every day, at three AM, the server makes a tarball of all relevant data and throws it into a special backup directory. A launchd-powered script on my Mac grabs the tarball daily and puts it in a place on my machine where Mozy can find it. The end result is a system I’m pretty happy with. I shouldn’t lose data that way again.

All that said, it seems to me like this situation shouldn’t even be possible nowadays. I understand that I should’ve been more careful; I won’t argue that. But…shouldn’t Unix have an undo command? I know for a fact that I’m hardly the only person to have hosed data by not paying enough attention while doing sysadmin tasks. Indeed, it’s regarded as a right-of-passage for system administrators, and focusing on the fact that this is the first time I’ve ever done something nearly so stupid on a production system makes me happy. But, still: on a Mac, or even on Windows, I have undo all over the place. I can’t think of any command on Unix that has undo. Isn’t it about time that started changing?

One of my friends adamantly refuses to carry a cell phone on him. Although I don’t have a lot of sympathy for that these days, I’ll be changing my opinion very quickly if blasting your friends starts to become common.

One of the Big New Features in WordPress 2.2 was a dynamic sidebar. The idea was that developers would write reusable Widgets that users could add to their sidebar through drag-and-drop—a huge improvement over the old method of modifying a bunch of PHP by hand. The good news is that building a sidebar from widgets works great. Unfortunately, the bundled widgets don’t. The archive widget has an invalid capitalization of its onchange event that keeps this site from validating, while the links widget serves up duplicate IDs if you have links in multiple categories. No software is perfect (sadly including my own), but there’s a difference between having a few random bugs, and failing profoundly at something integral to your product—in this case, generating valid XHTML. The second bug in particular is surprisingly severe, since having duplicate IDs breaks the DOM and violates not just XHTML, but also the generally more bug-tolerant HTML 4.

What I don’t understand in looking at these bugs is why they happened in the first place. HTML validation can be trivially automated on a build server; it’s equivalent to verifying that the code you check in compiles (something which indeed frequently is automated on build servers). Shipping validating XHTML is something that the developers should be able to get “for free.” Yet it’s apparently not part of their process. The bug for the onchange mistake, for example, didn’t get filed until July—several months after WordPress 2.2 shipped.

Even if HTML validation were part of their build process, I’m not sure it would have helped much: the duplicate ID bug was filed the instant WordPress 2.2 came out, but making a fix has been pushed to WordPress 2.4, due sometime in 2008. They had to do this because the underlying bug is actually in the core of WordPress, which means fixing it would be hard.

What good is knowing about bugs when you’re not willing to fix them?

Lotus Symphony is IBM’s rebranded version of OpenOffice, and ships with Lotus Notes. As of today, Symphony is free. Even if you have OpenOffice, Symphony may be worth checking out, as it sports what in my opinion is a superior interface.

I’m generally quite paranoid when it comes to server security—doubly so because I’m no guru at it—so I tend to take a shotgun approach. The virtual server running bit qua bit has a restrictive firewall setup, has root disabled, only allows secure IMAP/POP/SMTP, disallows password login through SSH, and mails me daily security audits, among other things. I also monitor Debian’s security-announce list like a hawk. (If you’re the sysadmin for a Debian server and you’re not on that list, sign up. There’s no excuse for not keeping your box secure when you can get told about all the known vulnerabilities.)

One of the things that’s crossed my mind in the past month as I’ve been whipping bit qua bit into shape is that the login pages for WordPress and a handful of other utilities I keep on the server are served over HTTP, when they ought to be served over HTTPS. A couple weeks ago, I made available a secure version of the login page for those who wanted it. (Because I don’t want to bother paying a CA, I left the default insecure, since you’ll get a Scary Dialog Box telling you to panic and fear because the world is ending and you are going to die and incidentally the cert’s self-signed, but nobody reads that far and fewer people understand what it means, so they just leave the site instead of posting.) A week ago, I moved all the back-end pieces so that they were available HTTPS only, and a few days ago, I decided to move the WordPress admin interface to HTTPS as well.

I naïvely thought the process would be simple. I fired up Emacs, opened the configuration file for bitquabit.com’s insecure site, and added the following line:

Redirect /wp/wp-admin/ https://bitquabit.com/wp/wp-admin/

I restarted Apache, checked things out, and bingo! Everything seemed to be working spiffy.

Except…I gradually began to notice that something wasn’t quite right. WordPress no longer automatically saved my drafts, and I couldn’t modify the websites appearing in my blogroll. When I found I also couldn’t upload pictures for an upcoming article, I had to dive underneath the covers and figure out what was going on.

It turns out that a lot of URLs in WordPress—among them, all the AJAX ones—are hard-coded to be HTTP, not HTTPS. When they were trying to execute, they’d get back a 302 response code (redirect), and then fail. Not good.

There are two ways to secure WordPress: the first is to simply modify the blog’s URL to be https://—not really what I was aiming for. The second is to use the Admin-SSL plugin so that only the login and admin pages are secured. That seems to work perfectly, but it forces all users to login through the secure interface, which means that everyone gets the Scary Dialog Box when they try to post. I’m not sure what solution I’m going to end up adopting; part of me still just wants to rewrite the whole blog myself, in which case I can easily customize it to do what I want, but it’d probably be easier to learn just enough PHP to fork the Admin-SSL plugin. For the moment, I’ll just keep using an SSH tunnel to the box when I want to post, which is probably more secure anyhow.

After reading about Michael’s attempt to turn his homepage into an aggregator for all his computer activities, I got inspired to try LiveJournal again. I last used LiveJournal when it was cool because it was running on Linux, and Linux was really cool because it had the singular ability to wipe out huge chunks of nominally backed-up data if you didn’t understand how UMSDOS worked. Since (as you “old-timers” have already figured out) I was thirteen at the time, I had of course forgotten my password, so I decided to make a new account. After a few moments reflection, I opted to name my new account “bitquabit.” I filled out the new user registration form, clicked a button, and started using my new account got a violent, red error:

Username already in use; please select a different one.

Now, I know that the name “bit qua bit” was new at the time I created this site, because a) I googled it thoroughly on all the major search engines, and b) most people I know who speak Latin can’t agree on whether “bit qua bit” is even grammatically legal, let alone what it means, so chances were low that someone just happened to come up with the name independently. Furious to find out what nefarious scoundrel would dare abscond with my originality, I rushed over to bitquabit’s LiveJournal and was greeted with…

…the headlines for my own blog, linked to my own blog.

It turns out that LiveJournal has a feature called syndicated feeds, whose sole purpose in life, as near as I can tell, is to let you friend arbitrary RSS and Atom feeds that aren’t on LiveJournal. I understand the underlying motivation, but I’m perplexed by their implementation. It wouldn’t be hard for LiveJournal to implement the same functionality without pseudousers; as-is, they’re conflating two radically different classes of accounts behind the same user interface. Even if, for some technical reason, they do need pseudousers, I don’t get what the point is of hosting an HTML version of the RSS feed on LiveJournal. There already is an HTML version of the RSS feed. It’s called bitquabit.com.

I’m not complaining, mind you; the way they’ve implemented the feature—just the headlines, with links to the full content—all it can do is boost my pagerank, and since I only wanted to log into LiveJournal to see what it’s like these days, I don’t really care that I had to go back and come up with a random user name. I just find the whole concept truly bizarre.

I barely ever visit blogs anymore. Instead, I tend to just read everything in the comfort of my news reader (Google Reader, if it matters). Nevertheless, I still think there’s a place for having a good website design, and bit qua bit’s stank like a disemboweled skunk in an outhouse. So, rather than sleep tonight, I’ve been working like mad trying bang out the last of my redesign before the holiday, and I’m happy to say that I’ve basically succeeded. There are a few things I don’t like, a lot that needs a bit of polish, and one or two bugs left, but, overall, I’m extremely pleased with the result.

Comments on what you like and what you don’t more than welcome; I’m quite willing to admit that web design isn’t exactly my strong suit.

Edit: Any IE experts want to help me figure out what magical incantation I need to say to make the design look the same in Explorer as it does in Opera, Firefox, Safari, and basically every other browser made except IE?

This morning, I’m at the big NYC FogBugz demo, the only part of the FogBugz World Tour I’m actually attending. Hopefully, I’ll see some of you there.

For the last several months, I’ve been powering bit qua bit’s mail system with Citadel. In the yonder years, Citadel was a very powerful BBS for Unix systems. As the bulletin board days drew to a close, and its developers began searching for a way to keep Citadel relevant, they hit upon the idea of turning it into a groupware system. The current version of Citadel runs on most Unix platforms, supports secure IMAP, SMTP, and POP3 out of the box, provides the GroupDAV protocol for synchronizing with calendars and address books, and comes with a robust web client that lets you access everything when you’re away from your laptop.

Citadel’s been simply wonderful. Installing it under Debian Etch was trivial, involving little more than apt-geting from their repository and answering a few very short questions. Integration with SpamAssassin involved little more than a few clicks, as do most administration tasks. Setting up restorable backups took all of five minutes, and making Citadel receive mail from multiple domains only took a bit longer. And, when I get really nostalgic, I can still connect to the system as if it were a text BBS to chat with users, perform administration tasks, or (not recommended) read my mail. (Use mutt if you want to go that route, as the Citadel text mail interface is gleefully unchanged from 1995 or so.)

Citadel’s hardly perfect. It gets a bit confused if your mail client tries to grab several IMAP folders at once; its web interface, though rich and functional, feels primitive; its sendmail replacement is still under-featured; and setting up aliases is more confusing than it has to be. Still, I think it’s worth a look.

My day job is working on Fog Creek Copilot, a powerful, cross-platform remote assistance solution. This week, Tyler and I were talking about how it’s too bad that Copilot didn’t really exist when we were in college, because we always ended up doing tech support for our families over the phone, which always went something like:

Me: What do you see now?

Family Member: A dialog box.

Me: What’s it say?

Family Member: It’s got a stop sign with an exclamation mark and says that the server can’t be found.

Me: Okay, click “Okay,” then read me back the line that says “SMTP Server.”

Family Member: Wait, I just clicked “Okay” twice. Now what do you want me to do again?

And so on. Not fun.

The good news is that Copilot exists now, and makes doing remote tech support really easy. Unfortunately, college students are basically perpetually broke. Tyler and I remember what that’s like. It sucks. You just got to school. There are eighty bajillion things going on and at least ten girls or guys that have caught your eye and (if you’re very lucky) your pants. Having to pick between wasting an hour helping someone, or using valuable booze money just so you only need five minutes, can be a painful choice.

Well, we’ve got a proposal: for the month of September, we’ll let anyone with a .edu address use Copilot for free, up to three times. These aren’t two-minute trials; they’re real, legit, 24-hour day passes. Plus, if that’s not enough, we’ve got a referral program: if you refer a friend with a .edu address to use Copilot, you get another three day passes. Ad nauseam. No limit. No catch.

So if you’re in college, the next time someone asks you for help, grab a free Fog Creek Copilot day pass. Then spend your leftover time grabbing a beer. Best of both worlds.

It turns out that you can have WordPress automatically show a post after a specific time. To do so, simply set the post’s time stamp in the future. Presto! The post won’t appear either in RSS or on the main page until after the time you’ve set, and in the management interface, shows up as a “scheduled post.” I’ve actually been using this feature for awhile now: I wrote most of this week’s posts on Sunday afternoon, but set their post times so that they’d appear steadily throughout the week. This lets me write when I feel in the mood, but end up publishing daily, even if I forget/don’t have time/don’t see anything interesting to write on that particular day. If only I’d known about this sooner…

From Slashdot, which is slowly redeeming itself, comes a link to Microsoft admitting that it bribed members of the Swedish ISO committee to vote for OOXML. Unsurprisingly, the Swedish ISO committee just voided its own vote. Due to time crunch, they will not be casting a vote at all in the Open XML ratification process.

I find it depressing but predictable that I’m unsurprised.

I think that the Wall Street Journal does a fairly good job covering technology from a consumer’s perspective, but I feel that they struggle whenever they try to cover more industry-focused issues, making outright mistakes and failing to understand what in the debate is actually important, which leads them to follow up (or fail to) on the wrong points. Today was no exception: in an article entitled “‘Office’ Wars,” they attempted to cover the politics revolving around Microsoft’s efforts to get their Open XML adopted as an ISO standard. The mistakes began cropping up depressingly early in the article:

To gain approval as an international standard, Microsoft had to bare the code that undergirds the Office file format, called Open XML.

Um, no. Microsoft created a brand-new file format called Open XML that is a totally different format from the ubiquitous DOC format. They then published a 5000-page specification of its supposed operation that is incomplete and inconsistent with their own Open XML implementation, which they have not had to lay bare. This not-actually-implemented-as-specified Open XML specification is the one Microsoft is trying to ram down everyone’s throats.

Jean Paoli, one of Microsoft’s top standards experts, says the company wants Open XML adopted as a standard to encourage rivals to use its format, not squelch interoperability. He points out that other vendors, including Apple Inc., are adopting it.

Apple supports reading an extremely limited subset of Open XML in TextEdit in its upcoming Mac OS X Leopard. The most recent version of Pages can also import a slightly larger subset of Open XML, but, so far as I know, can’t write it back out. Neither Pages nor TextEdit use Open XML as their native formats. I do not think that Apple’s behavior can honestly be construed as “adopting [Open XML].”

He says IBM is stirring up opposition to Open XML’s gaining approval from the International Organization for Standardization, or ISO, to protect its Lotus Notes office suite, which uses the rival format Open Document.

Probably partially true, in the sense that Lotus Notes bundles a version of the open-source OpenOffice, which uses ODF as its native format, but since there actually are several word processors that work natively with ODF, I’m hazy how this could be construed to protect Lotus Notes. In fact, given that Lotus Notes used to ally itself with SmartSuite, which had a proprietary file format, I think this actually opens up Notes to more competition. Given that there are no word processors other than Word that support Open XML, I think Microsoft’s claim applies far more strongly against itself.

Open Document is already an ISO standard, but Microsoft says there’s room enough for more than one document standard.

Why? “Just because” isn’t a good enough reason. ODF has already been standardized for some time and has broad industry support. Only Microsoft Office uses something superficially resembling Open XML. This strikes me as a hilarious extrapolation of the old joke, “The only problem with standards is that there are so many to chose from.”

In addition, Mr. Robertson said, the technical committees should include lots of voices—and that means some on Microsoft’s side. “Where you find expansion in the committees, you will find expansion on both sides,” he said. “That’s OK because it represents the community a whole lot better.”

This is the “all ideas are equally valid” fallacy. If we are going to have a debate on whether we should require, by law, that people dress up in purple chicken suits and make monkey noises at 3 PM on the second Thursday of the month, no one would be particularly surprised if a committee were completely biased against the idea. Even if I tripled the size of the committee, having a purple chicken suit proponent likely would actually make the committee less representative, since the position would then be over-represented.

We have a similar situation with Microsoft’s Open XML. IBM, Sun, RedHat, Novell, Canonical, and Google, among others, support ODF. So far as I know, the only major company backing OOXML is Microsoft. Why does it follow that we should expect roughly equal numbers of OOXML proponents and detractors on any given committee?

I appreciate that the WSJ has recognized that the OOXML-ODF debate is an important one, and I’m glad that they’ll be increasing public awareness of it, but I still wish that they’d done a better job covering what’s actually going on. This is a case where both sides are not created equal, and fair reporting probably should not treat Microsoft as if they have equal merit.

I’d sometimes like to think that I can be a purely rational person, but the fact is that I’m anything but. I’ll ignore software that does what I want if it’s not “pretty,” and I’ll often ignore software that does what I want and is pretty if it’s not “open enough.” I favor using the NYC Subway or walking instead of taking the bus, even when I know darn well the bus is the fastest option. I frequently ignore weather reports when I leave in the morning, instead going by what I feel the weather’s actually going to do.

My powerful distaste for MySQL is not irrational. The nominal database has corrupted the minds of a generation of web developers who don’t really understand ACID, are afraid of using transactions, couldn’t make a schema that used a composite primary key if their lives depended on it, have a genuine fear of triggers and views, and think this is a fine and SQL-compliant way to quote a database entity. I am happy to say that no project I have ever worked on that needed a relational database has used anything except DB2, PostgreSQL, or Microsoft SQL Server—all real, ACID, SQL-compliant systems.

Yet though my distaste for MySQL is not irrational, the dirtiness I feel whenever I use a program that requires MySQL…probably is.

This blog is powered by WordPress, a blogging platform written in PHP (which is in turn a story for another day). WordPress requires MySQL. This has never sat well with me. It’s the only thing on the entire server that requires MySQL, which in turn requires me to learn how to administer MySQL just so that I can have my blog on here. Because MySQL has Unicode issues, WordPress has Unicode issues, and because I’m one of those jerks who insists on writing “naïve,” “café,” and, on rare occasions, even pretentious crud like “reëxamine” and “rôle,” this isn’t just a theoretical problem: when I moved to Linode from Dreamhost, I had to try several times before I finally got all of the posts properly moved. Not a pleasant experience. Yet I have a hard time giving up WordPress. It’s so easy-to-use, and it just gets so much stuff right, that I can’t really justify using an inferior project just so that I can keep the data on PostgreSQL.

When Six Apart announced that Movable Type would become open-source, I thought that my prayers had been answered. Movable Type is a powerful, tested, feature-rich blogging platform. It has superb traffic analysis. It does a great job filtering spam. It supports OpenID. It’s easily themeable. And, of course, it runs just fine on PostgreSQL. This, I thought, would be perfect.

Turns out I was wrong.

My problems began with the install. Installing WordPress, provided that you have PHP set up correctly, involves the following steps:

1. Extract all the WordPress files into a directory
2. Go to the install page
3. Tell it where your database is
4. There is no step four

Installing Movable Type, at least for me, looked more like this:

1. Extract Movable Type into the cgi-bin directory on Apache
2. Move some of the Movable Type files into your static directory
3. Go play with permissions on the cgi-bin directory, cause apparently MT needs to write things there
4. Install DBI for Perl via apt-get
5. After trying for a few minutes to figure out why MT insists I haven’t installed DBI for Perl, remember that DBI needs drivers for its databases; install them also via apt-get
6. Allow apt-get to pull in roughly 30 additional dependencies
7. Enter my database connection information
8. Play with permissions again, because MT just set them to 777 on one of its directories
9. Log in and get notified that I still have bad permissions, apparently now in the static files directory

Once I got all of that sorted out, though, my hopes were high. After all, you only go through installs once, and the WordPress caveat, “provided you have PHP set up correctly,” shouldn’t be overlooked. It’s a biased comparison. What really matters is how the software performs and how easy it is to use.

The answers are, respectively, “Slower than a 70-year-old man trying to outrun a hoard of Playboy Playmates” and “not very.”

Movable Type is slow. Really slow. Operations that are virtually instantaneous on WordPress take a significant amount of time for Movable Type. Publishing, for example, which involves hitting “Publish” in WordPress and completes in about half a second, takes a couple of clicks in Movable Type (you have to republish your blog à la Blogger, you see) and takes a few seconds. I’d be more willing to put up with that if it weren’t for the fact that WordPress is running on my virtual host at Linode—a shared CPU system where I’m allotted 256 MB of RAM—whereas Movable Type is running on Vera, a server my roommate and I keep in our apartment, which has a blasé 2 GB of RAM and a RAID controller. PostgreSQL isn’t the bottleneck, nor is Apache; I checked. It’s MT all the way.

I haven’t totally written off MT4 yet, but my experience thus far has not been a good one. I’ll have to see if my opinion changes as I work with it more. I’m slowly overhauling the look of bit qua bit to get away from this professional and severe theme to one that’s gentler and more fuzzy, and will be implementing the theme in both platforms simultaneously. I’m also going to muck with Vera a bit more to ensure that something’s not badly misconfigured. But, at the moment, it looks like WordPress may be my blogging platform of choice for quite some time to come.

This afternoon, on a lark, I installed Conkeror, a Firefox plugin that makes Firefox look and act like Emacs. As far as these things go, I’m actually extremely impressed. A substantial number of Emacs commands are implemented—including the less-common ones, such as C-x h (select all), that most Emacs-style emulators seem to miss. Suddenly, navigating the web entirely by keyboard seems…pretty reasonable. If you’re either an Emacs or a keyboard junkie, check it out. You may really like what you see.

Today, I was going through the sizable network that my roommate and I have built up, compiling a sysadmin-style binder of addresses, MACs, components, operating system settings, and so on, and was surprised how many systems we actually have on the network just within the apartment. The following is a full list. What I find interesting is how these machines’ names at once make them more “interesting” than just being “the machine in the bedroom,” and also seem to reflect something of what was going on in the brain of the owner at the time, to the extent I can easily tell which machines are mine and which are my roommates.

Mungus/Pacifica

• Apple Power Macintosh
• Mac OS X 10.4
• 2x2.5 GHz PowerPC G5
• 2 GB RAM
• 410 GB storage
• ATI Radeon 9800 XT
• 1680x1050 20” LCD

Squeegee

• Nokia N800
• Official Maemo-derived Linux
• 330 MHz Texas Instruments OMAP2420 (ARM11 variant)
• 128 MB RAM
• 256 MB Flash ROM
• 800x480 4.1” LCD

Twoface

• Dell Inspiron 6400
• Ubuntu Linux 7.04/Windows XP Pro
• 1.67 GHz Core 2 Duo
• 2 GB RAM
• 80 GB storage
• ATI Radeon Mobile X1400
• 1680x1050 15.1” LCD

Blue

• Custom-built machine
• Windows Vista Ultimate Edition x86
• 2.33 GHz Core 2 Duo
• 2 GB RAM
• 400 GB of stoorage
• ATI Radeon HD 2600 Pro
• Dual 1024x768 LCDs

Haven/Gumbi Expandomatic

Apple AirPort Express, used nowadays purely for music broadcasting (though, as you can probably guess from the name, it was originally an AirPort network extender as well)

Wii

Our Wii. I wish there were a name to change what name the Wii self-identifies as, but there doesn’t appear to be.

NintendoDS

We have a DS and a DS Lite, though neither I nor my roommate play them much these days

Persephone

Our router, a Linksys WRT54G 8.0

Smear

Our PAP2 VoIP-to-phone adapter

Vera

• Custom-built machine
• Debian Etch
• 1.70 GHz Pentium 4
• 2 GB RAM
• 144 GB storage

Solomon (Retired)

Solomon is all but dead (the memory clip, located directly under the keyboard, broke from strain, and I can’t justify replacing the whole mobo on a six-year-old machine), and is being left to live out its remaining days in my climate-controlled office at Fog Creek for occasional use.

• PowerBook G4
• Mac OS X 10.4
• 667 MHz PowerPC G4
• 512 MB RAM
• ATI Radeon Mobile 7500
• 1280x854 15.1” LCD

Okay…who else is psyched about being able to buy an i.Beat blaxx?

The German company also seems very excited about their upcoming sun-colored version, the i.Beat jooz.

Edit: The company has renamed the product simply “blaxx.” You can still see the original name in Wired’s piece on the player.

I don’t know why, but recently, as my love of really low-level hardware and my desire for low-power, high-performance computing has increased, I’ve been researching all the old, famous CPUs and operating systems. I started over what I swore was going to be a computer-free vacation by delving into programming in assembly for 680x0 Macintoshes (during which time I fell in love with 68k assembly), then explored ARM chips, and finally somehow or another ended up at 4:30 AM on a Sunday working on an assembler and cycle-accurate emulator for the MOS 6502. Resources I’ve found useful:

1. Assembly in One Step, which provides a superb overview of the 6502 execution model
2. A list of 6502 opcodes, including their mnemonics, size, and how many cycles they should take to execute
3. WLA DX, an open-source cross-assembler for a bunch of old CPUs (including the 6502, 65C02, 65816, and Z80), which lets me check that I actually understand the documentation and am writing my own implementation correctly

When I have something worth looking at, I’ll post it. On a lark, I’m actually writing my assembler and emulator in Free Pascal, a superb open-source clone of Borland Turbo Pascal. I’ll also post about how that goes. (The short version: my memory is better than the experience, but both are better than C++.)

Ars Technica reports that used CDs are going to be subject to waiting periods and resale restrictions in Florida, Rhode Island, Wisconsin, and Utah. Ken Fisher writes:

In Florida, Utah, and soon in Rhode Island and Wisconsin, selling your used CDs to the local record joint will be more scrutinized than then getting a driver’s license in those states. For retailers in Florida, for instance, there’s a “waiting period” statue that prohibits them from selling used CDs that they’ve acquired until 30 days have passed. Furthermore, the Florida law disallows stores from providing anything but store credit for used CDs.

Billboard also has the story. For any of my old Duke Moot Court buddies who can’t quite believe this is happening, you can read Utah’s version of the bill. The bill begins by redefining pawn brokers to include any shops that sell secondhand merchandise (13-32a-102), and then enumerates what must be recorded at the time of transaction. To whit:

(1) Every pawnbroker or secondhand merchandise dealer shall keep a register of each article of property a person pawns or sells to the pawnbroker or secondhand merchandise dealer, except as provided in Subsection 13-32a-102 (17)(b) regarding secondhand merchandise dealers. Every pawn and secondhand business owner or operator, or his employee, shall enter the following information regarding every article pawned or sold to the owner or employee:

• (a) the date and time of the transaction;
• (b) the pawn transaction ticket number, if the article is pawned;
• (c) the date by which the article must be redeemed;
• (d) the following information regarding the person who pawns or sells the article:
  • (i) the person’s name, residence address, and date of birth;
  • (ii) the number of the driver license or other form of positive identification presented by the person, and notations of discrepancies if the person’s physical description, including gender, height, weight, race, age, hair color, and eye color, does not correspond with identification provided by the person;
  • (iii) the person’s signature; and
  • (iv) a legible fingerprint of the person’s right thumb, or if the right thumb cannot be fingerprinted, a legible fingerprint of the person with a written notation identifying the fingerprint and the reason why the thumb print was unavailable;
• (e) the amount loaned on or paid for the article, or the article for which it was traded;
• (f) the identification of the pawn or secondhand business owner or the employee, whoever is making the register entry; and
• (g) an accurate description of the article of property, including available identifying s such as:
  • (i) names, brand names, numbers, serial numbers, model numbers, color, facturers’ names, and size;
  • (ii) metallic composition, and any jewels, stones, or glass;

This law is completely asinine, inexcusable, and downright disgusting. It impedes the free market, effectively criminalizes what ought to be completely legal behavior, and has absolutely no positive benefit whatsoever to anyone except the recording industry umbrella organizations. Get this information onto the mainstream media so that people can start understanding what the RIAA actually stands for.

When I first started bitquabit, I wanted it to be strictly a technology blog. When people wanted to read something about Squeak or db4objects or Copilot, they could come here. When they wanted to read someone writing a meandering essay on farm subsidies and ethanol, they could go somewhere else.

That position is becoming increasingly difficult to maintain. On the one hand, technology is inextricably tied to certain political agendas that, I feel, must constantly be discussed—patents and copyright chief among them, but also such topics as freedom of speech or the ethics of invention. Yet these topics are, well, boring, because basically every single prominent tech blogger has exactly the same position on these issues: patents are too broad, last too long, insufficiently investigated for prior art by the USPTO, and, at least in computing, do far more harm than good; copyright is great, but in its current forms last too long, fails in its original purpose, and violates the Constitution; and the Digital Millennium Copyright Act and its siblings abroad are blatantly unconstitutional due to their de facto destruction of fair use, stifle free speech, and should be at best struck down and at worst kept from spreading to more countries. Because these positions are so widespread, they’re neither controversial nor insightful. Others expound these arguments far better than I and are more up-to-date on the latest goings-on in the beltway. In short, though these topics may be relevant, they are also well-covered and therefore not issues I feel compelled to discuss here.

That leaves The Ugly Topics—the topics nominally only distantly related to technology. Topics like the war, Sarkozy’s election, the heightening tensions in the last month in Israel, the rapidly intensifying atheist-believer debate, and global climate change. These topics, charged though they may be, have profound impacts on technology. Wars divert funding from general science and research, yet fuel technological innovation—provided that the technologies in question are usable for waging war. Sarkozy happens to be sympathetic to digital rights management software and does not seem to support the EU’s drive to force Microsoft to open its networking protocols and file formats. Israel is a miniature Silicon Valley and a major exporter of cryptography and encryption software. The atheist-believer debate has profound impact for artificial intelligence and, indeed, the purpose and future use of technology in general. Global climate change drives ever-more-powerful computing clusters in an attempt to simulate weather patterns, spurs green buildings and high-efficiency solar cells, and may usher in the return of nuclear power. Even if we ignore all of their greater significance, the topics are still relevant.

The problem, of course, is that I cannot discuss just the technology side. Some readers are already fuming that I can touch on the war and step right past all the people dying, others are annoyed that I’m ignoring the “interesting” parts of Sarkozy’s politics, and my good friends, who are well aware of my fascination with religion, are curious why I’ve never touched a single religious topic on this blog. These issues are all so big that I have to discuss parts of them that have nothing to do with technology if I want to cover their technological aspects at all. Yet by attempting to address any of these topics, I will divide my audience.

If I want to speak on these topics, yet doing so costs me readers who are interested in what I have to say on technology, does that make addressing these topics “bad”? Does my hesitancy, however brief, to speak my mind and lose those readers make me a coward? If I honestly care more for this particular blog to reach a technical audience than a political one, does that change the answer? If it did, would the answer change again if I were to split bitquabit into a personal blog and technical blog?

Where is the dividing line?

I don’t exactly consider myself a bad-ass system administrator. In fact, to be honest, I’m a pretty poor one. I like programming computers, not maintaining them, and the hoops that system administrators have to jump through to get everything configured and running smoothly give me headaches. Granted, machines under my dominion usually end up stable after a week or two of heavy dogfooding, and, so far as I know, no machine I’ve administered has ever been hacked (knock on wood), but administration is most definitely not my forte.

So when I decided to move to Linode, I did so with a little bit of trepidation. Yes, I wanted more flexibility than Dreamhost offered, and yes, I wanted full control over a server so I could run whatever I felt like, but at the same time, I was extremely nervous about whether I’d actually be able to get everything working properly.

Mostly, I ended up having an easier go of it than I anticipated. Debian 4 has been a great distro and was wonderfully easy to set up. sudo apt-get install libapache2-mod-php5 php5-mysql helped me get WordPress back up and running very quickly. Getting mail operational using dovecot and exim4 with TLS proved more of a challenge, but I eventually got that working too, and even figured out how to get my self-signed certs permanently accepted by Thunderbird and my awesome Nokia N800 (more on that some other day). But I had two major problems: I was getting identified as spam by all the major hosting providers, and I could not for the life of me get Apache virtual hosting doing what I wanted.

The first drove me completely bonkers: even though I verified repeatedly that I had set up valid SPF records in DNS, Gmail kept nailing everything I sent as spam with no explanation whatsoever. Nothing I could come up with helped. At the same time I was trying to get inside Gmail’s head, I was also trying to make bitquabit.com point to, well, bitquabit.com, but make code.bitquabit.com point to an encrypted site that hosted my Mercurial repositories. Try as I might, I simply could not make a configuration that did what I wanted. I ended up with all virtual hosts authenticated, or the code subdomain burning out, or Apache rejecting my configuration entirely. I finally caved in and asked Michael Gorsuch, our sysadmin, if he could lend me a helping hand.

Michael’s awesome. Rather than tell me he was too busy, which I would have completely understood, and rather than just pointing me to clear documentation, which is what I had hoped for, he actually wrote tutorials of his own for his website that explained how to do what I was trying to accomplish. After reading the tutorials and getting a quick lecture on PTR records, I finally managed to get everything up. Everything’s working perfectly. No more spam blocks, my repositories are live, and I even managed to cram a few extras features in there as well (specifically, authenticated WebDAV). I don’t know of any other sysadmins who would’ve taken the time to help me like that, so thank you, Michael.

All you readers, go check out his blog. It’s a pretty good read. (And after this weekend’s redesign, it’s pretty pretty, too.)

What’s left? I’m not completely sure. The immediate impetus for moving to Linode was to enable me to rewrite this blog in Squeak using Seaside and Pier or HttpView2. Long-term, I wanted to be able to clean up and begin publishing some small web apps I’ve written in Seaside. Regardless of what I end up doing, I’ll be sure to let you know as soon as there’s something to click on.

Since I moved from Dreamhost to Linode, Gmail thinks that all email coming from this domain is spam. As near as I can tell, my SPF records are correct, and exim is definitely not set to be an open relay. Does anyone know what might be up?

Edit: At a friend’s recommendation, I checked Spamhaus and friends to see whether the previous owner of bitquabit.com’s IP might have been a spammer, but it’s not on any of the lists.

I was extremely happy to discover today that Ambrosia Software has finished porting Introversion’s DEFCON to the Mac. DEFCON is a happy-go-lucky simulation of global thermonuclear war. Each player controls a collection of boomers (nuclear-missile-armed submarines), missile silos, aircraft carriers, and airfields in an interface highly reminiscent of NORAD as depicted in the movie WarGames. Over the course of the game—which can range from a few minutes to a full eight hours—players compete to disarm their opponents while inflicting the highest number of causalities possible, as measured in millions of deaths. Although arguably ghoulish, the game’s surprisingly well-done, from the intuitive controls to the fairly solid AI to the quiet weeping that fills your ears as the first ICBMs kiss the ground. It’s not going to displace the Empire Earth series as my favorite strategy game, but it’s a very nice diversion. You can grab a demo from Ambrosia’s website.

So Google already has my email. They already know what news I read and what hobbies I have. They have my essays, my portfolio, my photos, and even my encrypted bookmarks and passwords, not to mention my code, my data, my spending habits, and my readership.

This morning, I log into Google, and discover that they’re now willing to track everything I do at any time anywhere on the internet and show it to me in a cute and cuddly UI.

I’m getting to the point where I’m having serious trouble convincing myself it’s possible to have an irrational paranoia of Google.

I’m a diehard Emacs user. When I first get into the office, I fire up Emacs, then check my mail in Emacs, then update all of my source files using either the built-in Subversion bindings or a Cygwin shell via Emacs, and finally get down to coding for the day in Emacs. Windows and Mac OS X at times feel like just the kernel that allows me to run Emacs.

Productivity-wise, that’s actually a great thing. My work environment is basically identical no matter what machine I’m on, enabling me to focus on coding instead of trying to remember exactly what keystroke does build versus clean build versus debug build on which platform. Unfortunately, Emacs relies heavily on the Control key. On older Unix systems, Control was conveniently placed directly next to the “A” key, which made it easy to use without turning your hand into a compact pretzel. On modern Mac and PC keyboards, you’re out of luck: the Control key has been relegated to the bottom-left-hand corner of the keyboard, making it hard to reach without contorting your hand.

Thankfully, fixing this deficiency is a piece of cake. On the Mac, the process is a bit confusing, but fairly straightforward: simply open System Preferences, go to Keyboard and Mouse, pick the Keyboard Tab, click on Modifiers, and set the Control key to be Caps Lock and Caps Lock to be Control. GNOME users have basically the same process: they must click Settings, Preferences, Keyboard, go to Layout Options, and then toggle what key they want to register as their Control key from among several options. In both cases, the setting takes place immediately. You do not need to log out or reboot.

On Windows, thankfully, the process is considerably more straightforward. First, open regedit (Start, Run, type regedit) and navigate to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Keyboard Layout. (N.B.: Keyboard Layout, not Keyboard Layouts. They’re different.) Next, right-click in the values field, choose New, and then choose “Binary Value”. Name the new field Scancode Map. The Scancode Map is a little-endian—encoded field that allows you to remap arbitrary keys. The header is simple: the first word is the version (all zeros), the next word is reserved (all zeros), and the third word is the little-endian—encoded count of how many keys you will be setting, plus one extra for good luck. So, if you’re only swapping your Control and Caps Lock keys—two custom mappings—your binary field should read 00000000 00000000 03000000. The next entries are the actual remappings, encoded as key to map (two bytes) and key to map it to (two bytes). Looking up the scancodes for Caps Lock and Control, we see that the former is 0x3A and the latter is 0x1D. So, to map control to Caps Lock, and accounting for the little-endian encoding, the next two words should be 3A001D00 1D003A00. Finally, we have a one-word null terminator, all zeros. The final value for the Scancode Map binary value, then, is 00000000 00000000 03000000 3A001D00 1D003A00 00000000. Make sure you’ve entered the key correctly, then reboot your machine. Presto! Caps Lock and Control reversed.

Give it a try. Even if you don’t use Emacs or a similar Control-heavy editor much, you may still find the reduced contortion when using the Control key is more than worth it.

There are few people in my life I admire more than my father. My father is the one who taught me how to swim, showed me my first BASIC programming, painstakingly explained algebra to me in fifth grade when I got annoyed at the confusing methods our my math teacher was teaching us, and demonstrated to me the value of problem-solving in high-anxiety situations when I was four by locking me in the child restraint seat and leaving me in the garage. I still turn to him when I’m in need of advice, support, or a good laugh.

Those who know me well know that I loathe Valentine’s Day. Part of that is just me being bitter—I’ve been in a stable relationship on Valentine’s Day exactly once—but a lot of it’s also that I don’t quite get why, on February 14th, everything is just supposed to be so much more hunky-dory than any other day. It’s a day created by card and flower companies to make sure you’re buying their products year-round, since otherwise, they’d have to go from Christmas to Mother’s Day without any real sales. If you really want to do something that will be romantically appreciated, in my experience, you’re much better off doing something random and out-of-the-blue when it’s least expected and your S.O. is feeling down than on a day whose main icon is a naked kid with bow-and-arrows who wanders around shooting people in the name of love. It’s like we have a whole holiday for John Hinckley.

So a few days ago, I was on the phone, whining to my dad about throwing a Fifth Annual F–- Valentine’s Day party and trying to figure out my personal life. My dad listened thoughtfully, and then began explaining to me what I was doing wrong.

The problem, he said, is that I date girls serially.

Let’s let V be potential difference between dating girls and doing something productive, I be current of love, and R be resistance to current flow. The problem with serial dating is the same as with serial circuits. If resistance starts to increase, you’re stuck: because V = IR, and R is increasing, I must decrease to hold the equality. Worse still, because P = IV, you’re just not going to have as much power with the increased resistance. Note on the above diagram that current is a lousy 214 mA, and we’re only able to get 1.93 “jewels” from our relationships. This is even worse than it seems: because there’s only one path—through all relationships—you’ll end up spending the majority of your energy on the relationship with the greatest resistance, which is exactly the opposite of what you want to do.

Worst of all, if (horror of horrors) you actually blow out one of your relationships, all current stops until you can manually patch things up. Your love life will be at least momentarily in ruins.

Now let’s examine the case where you’re dating multiple girls at once.

Even before we try the (admittedly more complicated) calculations, we can already tell the situation has significantly improved. Because we’re dating in parallel, we compensate automatically for higher resistance. Even though Lisa clearly is just not putting out, the result isn’t the massive slow-down we saw before, but instead results in conservation of energy, as you expend less effort on a mostly dead branch and focus instead on more promising branches. Whereas before, Lisa sucked the majority of our energy, now Sally and Judy do—at 16.2 and 9 jewels, respectively.

A broken circuit also no longer really fazes us. In the case that one of the relationships completely evaporates (which, let’s face it, Lisa’s not heading in a good direction), we’ve still got other branches to take up the slack. Best of all, because Lisa was high-resistance anyway, her departure barely affected net current, which decreases from 3.1 A to 2.8 A—both radically higher than net series current.

And with that simple metaphor, I suddenly felt much better about how things are going in my life right now. Dads are awesome.

So, in summary, dating serially is for chumps. If you really want to have a better fail-safe, be less affected by resistance, and have a wonderful net increase in power, go for parallel relationships. It’s the only way.

Recently, on reddit, someone linked to a map of the US interstate system laid out “subway style.” Rather than including all the geographical features of the United States, the artist opted to realign everything on a relatively simple aligned grid, emphasizing the purpose of the system (“get me from here to there”) rather than the implementation (“via this bridge over this river, using this exit by this town”). The artist himself complains tongue-in-cheek about the complexity of the existing system:

You know, the Interstate System is a pretty incredible bit of infrastructure, but have you ever looked at a map? It’s all over the place! Did those civil engineers never hear of a ruler?

Perhaps such a map is a nice idea in practice, but in the name of becoming simple, the map loses so much information that it becomes nearly worthless. The elimination of any sense of scale leaves you no idea how long it will take to get between two points. The absence of any geographical information means you have no idea what conditions or weather you may encounter. The elimination of intermediate place-names means that you cannot get to a location not on the map, no matter how close it may be to places that are on the map, unless you already know the complex system that the simplified layout is trying to hide. In an attempt to isolate the reader from the complexity of the Interstate, the map has given up an overwhelming amount of what made that same Interstate useful to begin with.

This whole discussion may seem like an aimless rant. After all, no one is seriously proposing to replace our normal Interstate maps with this simplified design. Yet such systems already dominate major mass transit maps. Take a look at the maps for the Chicago “L”, the DC Metro, or the T in Boston. The maps all list place names, but street names and major landmarks are completely missing (except for the Loop insert on the Chicago map), and times can vary tremendously. (Stops on the DC map, for example, go from at least one minute to about five with no indication, and relative distances on the map can be horribly misleading. Judging by the map, Foggy Bottom to Court House should be the same time as Metro Center to Farragut North, with both among the longest inter-station travel times in the system. That’s not even close to being correct.)

The New York Subway map is better in some ways and worse than some ways than Chicago, Boston, and DC. Taking advantage of the fact that most parts of the city are on grid systems, the map emphasizes intersections over place names. Although the map still lacks any meaningful scale, the grid system is simple enough that even a newcomer to the city, with a minute or two instruction, could make at least a rough guess for travel times—at least in Manhattan north of Houston. Even here, though, the map falls short. If someone wants to get to downtown Manhattan or one of the outer boroughs, they’re stuck. The map does not include enough information to make rational decisions without consulting an additional, detailed, scale map, just as with the other systems.

Now take a look at onnyturf’s map of the New York Subway. The subways and their stops still stand out and are easy to find, but now the entire map is to scale. The lines no longer run graceful curves; instead, the small blips and squiggles of long-forgotten zoning fights and long-gone support pillars for defunct skyscrapers are there for all the eye to see. Yet, if anything, the map has actually become easier to use. Someone completely new to the New York Subway could make a decent guess about how to get from one point to another, even if he were trying to get to a place not on the official New York Subway map. The added complexity actually simplifies the utility.

Simplicity is a good thing, but the focus should always be on simplicity for the user, not of the item itself. Sometimes, making something simple to involves exposing its warts.

Is it just me, or does anyone else enjoy pronouncing MySQL as “My Squeal”? (And of course, PostgreSQL as “Postgres Quell” makes a nice parallel, and is actually a nice hat-tip to the QUEL language Postgres originally used.)

I’ve also noticed that if you call Microsoft IIS “Microsoft Ice,” then the increasing rise of Mozilla Firefox seems to kind of balance things out.

I say nothing that should surprise diehard Mac users if I say that Mac OS X lacks decent power-user mail clients. Thunderbird suffers from a lot of the same problems as Firefox, Entourage is…well, Entourage, and Mail, pretty though it may be, is heavily underpowered in a lot of key areas. (E.g., you can’t even create nicely formatted lists, which is something I need to do quite frequently.) From this motley crew, I’ve traditionally opted to use Thunderbird. Yes, it’s ugly and doesn’t integrate with OS X, but it’s got the features I want, it runs quite fast, and I get to use the same app on both my Windows and my Mac machines.

Like a lot of people, I had been keeping a few thousand emails in my inbox to be sorted out at some point in the future (usually, “tomorrow”). Some of those were useless spam, and many were just trivial notes I should have deleted but didn’t, but a few were more important to me, like emails from my trip to Germany back in eleventh grade.

This morning, I brought my G5 out of sleep and noticed that Thunderbird had locked during the night. No biggie; force-quit and reopen. As I heated some tea and prepared my breakfast of burnt eggs cereal with spoiled milk instant oatmeal and a grocery list, I sat down to read my email, and noticed that Thunderbird was reporting that I had zero messages. It took my brain a few moments to fully process that Thunderbird was telling me that I had no messages, period. Thankfully, Thunderbird stores its emails in mbox format, and I actually do back my data up fairly regularly using rsync, so I didn’t worry too much, but I was fairly annoyed. The only other app that has ever lost my mail is Microsoft Outlook Express 4 for Mac—not good company to be in.

When I got home today and actually had a chance to look at the problem, though, things were weirder. The file for my inbox was not only fully in-tact, but Thunderbird was dutifully appending new messages to the end of the file. It simply refused to show me what it had downloaded. A few Google queries later, I had identified the problem: Thunderbird’s index file had gotten corrupted. The solution was to delete the index file.

This is dumb in so many ways I don’t even know where to begin.

1. My biggest complaint here by far is that Thunderbird has all the information it needs to fix this problem itself. It knows how many emails used to be in the account the last time the program ran. It knows how many emails the index file thinks there are now. If these numbers don’t match, rebuild the index.
2. It turns out that this is a known problem if your inbox gets big and you don’t compact it. What does compacting mean? It means actually deleting on disk what you deleted in Thunderbird. By default, Thunderbird simply marks certain messages as deleted, meaning that they won’t show in the GUI, but doesn’t actually expunge them from the file on disk, since that would require rewriting a large amount of data. Lots of programs use this shortcut for normal use, occasionally running a compacting cycle when the machine’s idle or at least not terribly busy. What’s unique to Thunderbird is that Thunderbird, by default, won’t compact, ever. That would be ludicrously stupid, but at least vaguely pardonable, if Thunderbird worked okay in this configuration, but the truth is that Thunderbird’s indices get corrupt if you don’t compact them often. To even discover this setting, let alone change it, you have to go to Tools:Options:Advanced, where it’s notably under “Disk Space” and not “Settings That Can Crash This App And Destroy Your Emails If You Don’t Change Them.”
3. Normal end-users are never, ever going to realize that they have any hope of recovering their data. To your normal end user, when the indices get corrupted because all that image-based spam they’re deleting doesn’t actually get deleted and corrupts the Thunderbird database, their email just vanished. Not what I’d expect from an application that bills itself as “a robust and easy to use client”.

At this point, I’m seriously considering reviving my earlier attempt to write a cross-platform email back-end in Squeak (or, these days, maybe Factor) with powerful threading and search, and then writing a nice GUI over it in Cocoa and .NET for normal use. In the meantime, I’m migrating back to Mail. It may be underpowered and it’s IMAP support may be an exercise in frustration, but at least its never deleted my email.

Sometimes, I have to wonder whether computers have a sense of humor. Yes, technically, it is a discount, but I think Amazon might want to consider not displaying the discount if it’s under, say, five cents.

A couple of years ago at Duke I served as an undergraduate teaching assistant, or UTA, for CPS108, which is basically the large-program design course. While there, I wrote a program called Crystal, which was a plugin-driven web browser written in Java. Basically, the code students received provided a couple of interfaces and a single concrete class called Crystal that implemented just a window frame and a handful of events to notify notify plugins of changes in the current URL viewed and so on. The students had to write the display components (at least an HTML view and a text view) and application services (URL bar, search bar, back and forward buttons, and so on) that would turn this shell into a real web browser. At the time, due to how close we were to finals and how much material we were trying to cram into the semester, the professor opted not to use Crystal, and I basically forgot about it. In the last semester or two, though, Crystal, now known as Websta, has made a reappearance.

While talking to Ben, a friend of mine who’s currently a UTA for 108, we got to talking about how to add tabbed browsing to the Websta framework. Ben insisted it was convoluted; I insisted that it was straightforward. (For the record, Ben was basically right; the interfaces provided are sufficient to allow for tabbed browsing, but students basically have to throw out the Crystal class and write their own shell in order to achieve a sane tab implementation.) In order to figure out how to do it, I asked Ben to send me a tarball of the current version of the source code. A moment later, I extracted the tarball, dragged the folder to TextMate, and was immediately greeted with this:

I spent a good minute or two frantically trying to figure out how to customize TextMate backgrounds before I finally realized that the TextMate icon in my dock had mysteriously changed as well:

It turns out that Allan Odgaard, TextMate’s author, pushed a special Halloween update into the bleeding-edge stream. Once I figured out what was going on, I had a good laugh and got back to work. I know that doing stunts like this would make a marketing department’s skin crawl, but I love how much personality little actions like this add to applications and the companies behind them. Somehow, it just makes things feel more personal.

I think it’s for that reason that I was so profoundly hurt several months ago when I learned that the Malkovich easter egg in Copilot had been removed. When doing development on Copilot, we usually run the Host (the helpee’s program) in a VMware session and run the Helper directly from our desktops. Occasionally, though, we need to test running Host on our own desktops (possibly because we need to test dual monitors, or because we want to see a Windows 98-to-Windows XP session, or because we really need the more powerful debugging facilities that VisualStudio provides to locally running applications), which results in a Copilot-into-infinity session, wherein each Copilot window contains another Copilot window which contains another Copilot window, etc. One day, Joel was looking at that and commented how similar it was to the movie Being John Malkovich when John Malkovich accidentally enters his own head and sees only other copies of John Malkovich. A few beers and a movie viewing later, Copilot had its first (and to date, only) easter egg: if the Helper and the Host ran from the same computer, the title bars would change to read “Malkovich.” I was especially proud of the code for this patch, which was, in its own little way, worthy of at least the lesser entries in the IOCCC:

void ClientConnection::Malkovich() {
    MALKOVICH MALK0VICH MALk0VICH
        MALKOV1CH MALK0V1CH MA1KOVICH
            MA1KOV1CH MaLKOVICH MALkOVICH
}

Sadly, the Malkovich easter egg, as software easter eggs are wont to do, was responsible for a bug in Copilot, and so had to be removed, but for those of you who really just gotta see it, fire off an email and I’d be happy to send you a copy of the Copilot helpers that still have the Malkovich feature in them.

You may have noticed that I’ve been modifying a lot of old entries recently shortly after I post just to twiddle some line breaks. That’s because Firefox 2’s rich text control is horribly broken. Now, I do recognize that a lot of the rich editors are driven by custom JavaScript, but a lot of the problems seem common to all websites. In no particular order:

1. Returns sometimes inserts BR, and sometimes inserts P, there’s no reliable way to tell which is going to happen, and they sometimes look the same based on nearby formatting;
2. If the last word in the previous line is italic, and you switch to italic, and then type a letter, the letter gets appended to the previous line;
3. In circumstances I haven’t entirely been able to sort out, the point is sometimes only half drawn, and either flickers between its upper and lower portions or only draws the top half; and my favorite,
4. Sometimes the point on screen has absolutely nothing to do with where text is actually going to get inserted, appearing randomly in the text, over a random part of the page that may be entirely outside the editor, or even just not at all

Combine all of these misfeatures, and you make it painfully difficult to write blog posts that actually show up on the main page the same way they show up in my browser. It’s common for an article to look just fine in the composer, but to show up on the website thoroughly borked. Yes, I should be checking WordPress’ preview more carefully, but I also shouldn’t have to deal with this problem in the first place. I’ve reverted to composing everything in TextMate in Markdown and then just pasting the raw HTML. There is no reason that things need to be this way; every single platform that runs Firefox has built-in rich text components, and it’s not that hard to convert from RTF to HTML. All Firefox has to do is bow down, use native widgets, and then convert on form upload.

Oh well. Maybe someday.