About once a week, I get an email in my mailbox that reads like this:

Hey, Kiln looks neat, but Git is totally the bee’s knees, so why the fuck are you using Mercurial?

Note that these emails are rarely (if ever) actually interested in why Kiln chose Mercurial; what they’re instead interested in is trying to piss me off enough that I get into a flamewar about why Mercurial is going to bring about Nirvana while Git causes people to eat babies using nothing but A1 sauce and a spork.

This is stupid.

Mercurial and Git are both DAG-based DVCSes. They use the same patch format. They both handle directories implicitly. They both can autodetect file deletions and renames. They both run on just about every platform I can think of. They both do a bunch of “cool” stuff, like rebasing, editing history, signing changesets, and serving as the inspiration for Ferris Bueller. They both have nearly identical performance characteristics, they both have social code sharing sites, they both are used by really big projects, they both have really good documentation, and they both got compared to James Bond or MacGyver or something like that in an analogy I didn’t really follow.

So why is there so much hating? I think what’s going on is that people are coming to these tools from Subversion or CVS, have their massive epiphany on how totally awesome DVCSes are, and then assume that only their tool can have this level of awesome, so they begin evangelizing. The problem, of course, is that the other guy feels the same way, and is also evangelizing, so the Git and the Mercurial guy end up in a locker-room-style temper-tantrum over whose tool has the best performance or whatnot, instead of how much more awesome their tools are than the competition.

This has to stop.

Mercurial’s enemy is not Git. Git’s enemy is not Mercurial.

Their enemy is Subversion.

For example, take a look at this tripe. Anyone who has seriously used Mercurial or Git for any length of time is going to spend most of the video alternating between laughing, peeing their pants, and trying to explain that they merely spilled a bunch of water on their crotch, honest. While there’s some truth to the video—a lot of people do ditch Subversion for off-line commits or for shelving or the like—the video also totally misses why people love and stay with DVCSes, which is that they make branching and merging actually work like they’re supposed to, and make source control so fast and seamless that you find you’re suddenly using it for everything.

But if you show that video to a Subversion user, they’re going to nod. And there’s really no reason for them not to: if you don’t grok what the DAG gives you, if you’ve never kicked an entire branch back-and-forth across a LAN, if you’ve never used a site like Bitbucket or GitHub, then the only tangible benefits you see from DVCSes are…well, partial commits, and the fact that their “checkouts” aren’t littered with .svn directories all over the place. And these are problems that Subversion’s upcoming versions actually might solve.

So it’s time to focus on the real enemy: the holdouts still using centralized systems. The way I see it, there are three parts to this:

1. Git and Mercurial need to do a better job handling one thing that Subversion is still better at: binary files. I know there are Git projects that are working on improving the transfer and storage of binary files within Git’s existing repository format, such as git-bigfiles. Mercurial is taking a slightly different tack through projects such as bfiles, which aim to deliberately move large binaries out of the store. We need to improve these workflows so that “DVCSes are totally better, unless” becomes simply “DVCSes are totally better.” (For our part, Fog Creek is helping fund development of bfiles via UCOSP, a semester-long student project.)
2. Git and Mercurial advocates need to remember that they need to be converting Subversion, CVS, and Perforce users, not each other. The fundamental evil here is centralized, branchless version control systems that effectively encourage all development to occur in trunk, and which make propagating bug fixes and features properly borderline impossible for all but the most disciplined shops.
3. Git and Mercurial advocates need to remember that anyone going to either system is a win for the both communities. It’s easy, in the yin/yang of Hacker News and proggit, to forget that most developers are not even aware of what DVCSes are or what they do. Yeah. Sounds crazy, I know, but trust me on this. The goal right now, if you honestly believe that the DVCS workflow is better—and I do—should be to get the mindset out there, to make more people aware of what DVCSes have to offer and why they should be using them. I for one definitely do not care whether you end up deciding that Mercurial is better than Git for you or not (well, I kind of do, because I want you to use Kiln, but otherwise…), but I get a warm fuzzy feeling knowing that, if I ever have to work with your code, I’ll be able to use a sane version control tool.

So let’s do it. No more fighting. Git, I hereby acknowledge you rock. And Mercurial, you rock so much I helped build an entire product around you. You’re both awesome. So you two shake hands…very nice. Now, see that other dude over there? The one with no real tagging or branching support?

Let’s get him.

This article is a continuation of Kiln’s Evolution, Part 1: DVCS as Code Review.

In the fall of 2008, Joel was getting increasingly adamant that FogBugz needed source control integration, and most people in the company seemed to think Subversion would probably be the best SCM to make that happen. Tyler and I disagreed, believing strongly that we should use a DVCS instead, and that our code review tool gave a really compelling example of why DVCS was better that any software shop would instantly “get.” But to convince the rest of the company, we’d have to show them a version of our tool that was more polished and usable than what we’d submitted to Django Dash.

And so we began a skunkworks project.

Unable to use time at work on much besides Copilot, I instead used my week of Thanksgiving vacation cleaning up the prototype’s user interface and functionality, named the result Kiln, and gave it its first logo. Tyler spent evenings in December making Kiln a proper, pluggable Django application, made the UI actually usable, and fixed a pile of bugs that would have blown up in our face if we’d tried showing Kiln to anyone else. By January, 2009, Kiln was ready to demo.

Kiln’s interface, directly after the winter skunkworks changes.

After lunch on a cold winter day, Tyler and I dragged everyone out into the kitchen and demoed the current state of Kiln. We showed repository management and the FogBugz-inspired code review workflow, and then made the case that this, or something very similar, should be FogBugz’ source control system.

And an amazing thing happened: somehow, everybody basically agreed. Sure, some people thought Kiln should be in C# or Wasabi instead of Django, no one could agree on whether Kiln should be a direct part of FogBugz or be an independent product, and Tyler and I argued strongly for Kiln as a hosted-only solution to a bunch of people who knew their bread-and-butter came from licensed applications, but everyone agreed that the basic of idea of a Mercurial-powered SCM with DVCS-backed code review made for a compelling product. And so Kiln was born.

For Kiln to develop into an adult, though, we had to assemble a Kiln team. Tyler and I were still working on Copilot, and the newly appointed team lead, Ben Kamens, was busy with the FogBugz 7 release. Even if all three of us started work immediately, we couldn’t possibly turn the project from prototype to beta by our target date of August 2009, and starting immediately seemed…well, optimistic, at best.

But we work at Fog Creek, and if there’s one thing Fog Creek knows how to do, it’s how to help interns churn out awesome products over the course of a single summer. After all, Tyler and I started at Fog Creek by developing all of Copilot in the summer of 2005; why not go for broke and try for a repeat? What we therefore decided to do was to bet the farm and put all of our summer interns on Kiln. The three of us would try to wrap up the work we had to do on our current projects as quickly as possible, and, as we transitioned off, we’d focus purely on building up enough Kiln infrastructure that the interns could immediately be productive when they arrived. Meanwhile, until the three of us could start work on Kiln, we’d have our project managers figure out the details of the user experience so that, once we finally could work on Kiln, we’d be able to focus as much as possible on coding instead of decision-making meetings.

As we moved ever closer to June, we made several key decisions about the design of the product:

1. Kiln could launch hosted-only, but we’d need to ensure that its design was amenable to on-site installation.
2. Kiln would depend on FogBugz for user management and bug-tracking integration, but would otherwise be its own code base.
3. Kiln’s website would be written in C# and ASP.NET MVC, completely freeing it from the FogBugz legacy code base.
4. The part of Kiln that needed to talk directly to the DVCS would be a separate component so that we target different (or even multiple) SCMs without changing the website.
5. Code reviews on branches would be eliminated in favor of arbitrary discussions on files and changesets.

By the time the first interns arrived, we had a beautiful set of specs with lovely Balsamiq Mockups put together by Jason and Dan, and we’d managed to cobble together a basic framework that supported repository hosting and FogBugz integration, and that learned as much as possible from our best example of a known-good ASP.NET MVC code base, StackOverflow.

Our plan paid off ridiculously quickly. A week into the internship, the interns had already managed to get key pieces of Kiln limping along. By the end of the second week, they had enough ownership they were starting to challenge us when they felt the user specs or the engineering didn’t make sense. Their strong focus on core Kiln freed Tyler, Ben and me to focus on performance, billing, On Demand integration, and all the other things that absolutely must get done for a real product, but that no one would otherwise ever do.

Just over a month into the summer, Kiln might not win any speed or beauty awards, but nearly all of its features were working in one way or another, and it was usable for its intended purpose. In other words, we’d hit pre-alpha. With great fanfare, we decided that Kiln was ready for dogfooding, and Kiln development moved to Kiln itself.

There’s a slightly unfortunate thing about dogfooding, though: features that looked great on paper, and even worked perfectly in the prototype, end up not being what you want in the real product. Some interfaces end up not scaling the way you want. Some end up too complicated, or end up solving one particular problem at the expense of all others. It’s a testament to our PMs that we had comparably few of these occur, but sometimes the difference between the prototype and what we ended up shipping was massive. For example, compare the Balsamiq mockup of the code review system, which was actually used by the pre-alpha:

with the version that we ended up actually shipping:

Or take a look at the original specification for the Kiln Dashboard:

compared to the shipping equivalent, the Activity Feed:

(I apologize about using the mockups, rather than screenshots, for the earlier versions; trying to get Kiln circa June 2009 running at this point proved a royal pain in the butt, and I don’t honestly think that it makes a big difference.)

What’s not obvious in these two screenshots is that the change from the pre-alpha interface to the shipping interface frequently happened over the course of just a couple of weeks. Everyone was very vocal about what they liked and didn’t like, and the interns were happy to go through several iterations rapid-fire to find one that everyone liked. In that way, the weakest parts of Kiln ended up getting the most attention, and rapidly matured into some of its strongest features.

In a massive code sprint at the end of the summer, Kiln matured into something resembling a fully grown product. One of our interns made a beautiful JavaScript renderer for Kiln’s DAG, the novel repository management our PMs designed fully matured into beautiful JavaScripty goodness, our FogBugz/Kiln workflow became increasingly seamless, and we further loosened review requirements so that you could review arbitrary discontiguous changesets. We transitioned Copilot and FogBugz to be hosted on Kiln as well, got mostly positive feedback from the rest of the team, and worked on swiftly addressing their complaints. Despite these feature additions, Kiln’s performance went from tolerable, to better, to fast. We knew we had a winner on our hands. We prepared the FogBugz On Demand environment to become Kiln On Demand, made our first deployment, and turned the switch for our first batch of beta users.

And while you might expect this part of the story to be about everything going haywire and all hell breaking loose, what actually happened is that, against all odds, everything basically worked. The beta was quite boring: while there were a lot of bugs to fix at first, and some of them were extremely tough (for example, supporting very large repositories, or making history views faster, or legitimately supporting Internet Explorer) or really ticked off our customers (Kiln at once point let you rename and move repositories without breaking URLs, which sounded like an absolutely great idea when I helped hammer it through the design committee, but which completely blew up in one of our client’s faces a few weeks later), everything basically worked. Our beta testers seemed increasingly excited about the product. All of our gambles seemed to have paid off as we readied Kiln 1.0 for its November launch date.

Except, of course, that Kiln did not ship in November, 2009. That’s because, just a week or two before it was supposed to ship, we went to the Business of Software conference in San Francisco.

You see, that was where we realized we were doing it all wrong.

To be continued…