Fresh graduates often push for a rewrite at the first sign of complexity, because they’ve spent the last four years in an environment where codebase lifetimes are measured in weeks. After their first unsuccessful rewrite they will evolve into Junior Engineers, repeating the parable of Chesterton’s Fence and linking to that old Joel Spolsky thunkpiece about Netscape³.

Be careful not to confuse this reactive anti-rewrite sentiment with true objections to your particular rewrite. Remind them that Joel wrote that when source control meant CVS.

…

3. The real reason Netscape failed is they wrote a dreadful browser, then spent three years writing a second dreadful browser. The fourth rewrite (Firefox) briefly had a chance at being the most popular browser, until Google’s rewrite of Konqueror took the lead. The moral of this story: rewrites are a good idea if the new version will be better.

The Joel article bugs me for a number of reasons, and the fact that his core example is totally demonstrably historically wrong is one of them.

Rewriting software is something you should only do if you can answer the question, “Why will it be better this time?”

New programmers to a code base think the answer is “because the last people working on this were idiots.” Sometimes that’s true! If you’re inheriting, e.g., some PHP code written by people who weren’t really web developers but just learned by copy-pasting, you might be smarter-enough to have a rewrite work. However, that’s the exception, and typically, you’re not any smarter than the people who wrote the code in the first place, so you’re not going to do any better.

Another answer that’s possible but unlikely to be correct is “they made bad technology choices.” Yes, some technology choices really are so bad that correcting them can make a difference, but moving, e.g., from Rails+MySQL to Node+Mongo is unlikely to make things better.

A good answer is “the business requirements they built this product for no longer apply.” For example, Microsoft Office vs. Google Docs or iTunes vs. Spotify. The former product in both cases is more robust, more complete, technically capable of doing it all, but the latter product by virtue of not having to do things that are no longer necessary can be radically simplified.

Anyway, John Millikin is correct: no haunted forests!

My friend and former colleague Jason Goldstein has a great article up about the problems with Python’s asyncio framework.

For what it’s worth, when I was at PBS, a different coworker and I tried to do a test project to learn how to write asynchronous code. We wrote scripts in both Python 3 and Go that would go onto Github, get a list of users on our project, and download their personal repo information concurrently. When we finished, we compared the apps to see the strengths and weakness of the languages.

Both apps ended up working (although the Python app cheated in a few ways, for example by ignoring paginated responses), but I found the Go app to be easier to write than the Python app, even though it was significantly more verbose. One of the biggest problems for the Python app was just finding documentation that I could understand and apply. In Go, the main problem was that you’re writing the concurrency scaffolding yourself, so it’s easy to write a spaghetti mess if you let yourself. In Python you more often run into the problem that doing something concurrently is a pain, so you do it in a blocking manner even when you shouldn’t. For example, really you should be collecting links asynchronously and adding new links to a queue as you go, but it turns out to be easier to do things one at time, even if that’s less efficient.

Camille Fournier has a great article on How do Individual Contributors Get Stuck?, but I’d like to focus today on the opposite question: How do programmers get into flow?

My theory is that there are three different idealized programmer personality types, each with its own strengths and weaknesses. Before you can get into flow, you have to know what kind of programmer you are, so that you know what kinds of things put you into the flow. Not everyone has the same strengths, and that’s okay. In fact, for a good team the more diverse your strengths the better_!_ If everyone on the team has the same strengths, that means they have the same weaknesses and the company may have major blindspots as a result. So what are some strengths and weakness of different kinds of programmers?

1. Programmers who just want to get things done

   Some programmers are happiest when they can see concrete results of their work. Sure, the code they end up producing has some rough edges, but they made something you can see and touch today. These programmers get frustrated when their work has no visible output and seems like it’s just rearranging code that already works.

   • Pros: Work fast
   • Cons: Work can be sloppy and hard to extend
   • Bored when: Refactoring or stuck on a hard problem
   • Best use: Getting the prototype 90% done in one week (the other 90% will take a team of a dozen six months)
   • Spirit Languages: PHP, JavaScript

2. Programmers who just want to solve hard problems

   For some programmers, the harder the problem is, the harder it is to resist. Sure, they were supposed to change all the borders from light grey to dark grey two weeks ago, but they got a little distracted by implementing a ground up rewrite of the rendering engine in OCaml and assembly language, so that will have to wait.

   • Pros: Will solve a problem you never thought possible to solve
   • Cons: What they produce can be hard for anyone else to grok
   • Bored when: Tediously gluing bits of code together
   • Best use: Creating the core algorithm that no one else on the team understands but gives your company the edge
   • Spirit Languages: Haskell, Clojure

3. Programmers who just want to create the perfect thing

   Some programmers are obsessed with beautiful code. They can’t believe that the code base uses CamelCase and snake_case in the same project. They’re personally offended that a method call runs in O(N²) time instead of O(N log N). They’ve started a secret branch of the repo to fix all the whitespace problems and rewrite the API, but they have to do it on their lunch break because the client just wants to ship a new feature.

   • Pros: Create maintainable, performant code
   • Cons: Prone to bikeshedding and wheelspinning redesigns; suffer from fear of a blank canvas
   • Bored when: The inherited codebase is ugly but there’s no time to fix it
   • Best use: Refactoring work done by the other kinds of programmers; bug hunts
   • Spirit Languages: Python, Go

Of course, in reality, no one is really just one kind of programmer. We all mix and match different personality elements on different days. But it could be helpful to know yourself and know what kind of programmer you tend to be the closest to in order to utilize your talents best.

So… what kind of programmer are you?

Here is a timely post by Bill Sourour called The code I’m still ashamed of:

If you write code for a living, there’s a chance that at some point in your career, someone will ask you to code something a little deceitful – if not outright unethical.

This happened to me back in the year 2000. And it’s something I’ll never be able to forget.

…

As developers, we are often one of the last lines of defense against potentially dangerous and unethical practices.

…

The more software continues to take over every aspect of our lives, the more important it will be for us to take a stand and ensure that our ethics are ever-present in our code.

Since that day, I always try to think twice about the effects of my code before I write it. I hope that you will too.

I think most software engineers want their software to help people and are very concerned by the idea that it might hurt someone. For example, a developer I know is working on a scholarship application form, and he expressed anxiety that a bug in his code might cause someone to lose out on thousands of dollars in tuition money. Consider how much more concerned we should be when hurting people is not an unfortunate bug but the goal of the software in question.

The title of this blog is a bit of ha ha only serious, but given the state of the world at present, now is the time to really think as an industry about the serious part of it.

Maciej Cegłowski has been calling on developers at large tech firms like Google and Facebook to think seriously about how their behavioral data could be used to hurt people, and asking employees to push for better protection of users’ privacy.

I think asking these questions is very important. It’s also very important as individuals to remember that you have a choice. If your company asks you to do something that hurts people, you can say no. I can’t promise you won’t lose your job, but making sure that you are helping people and not hurting them is more important.

You can find yourself another job.

You can’t find yourself another soul.

In the words of Solzhenitsyn:

You can resolve to live your life with integrity. Let your credo be this: Let the lie come into the world, let it even triumph. But not through me.

Addendum (Nov. 18): Please consider reading and signing the Manifesto for Responsible Software Development.

Everyone knows that debugging is twice as hard as writing a program in the first place. So if you’re as clever as you can be when you write it, how will you ever debug it?

— Brian W. Kernighan

The other day at work I was trying to write a test using VCR, a popular Ruby tool (with ports to many other languages) that records and plays back HTTP calls to external services, so you can test your client without actually pinging the external server more than once. I ran into a problem, however. No matter what I did, VCR refused to record my test. Was it because I was testing a POST request instead of GET? Was it because I was using a file attachment? Were there bad file permissions on the cassette directory? Was it not properly hooking into my client’s HTTP library? I spent the better part of a Friday trying to solve the question of why VCR wasn’t recording my request and response.

On Monday, I sat down quickly found the problem: I had reorganized some of the code before writing the test and it was failing before a request could even be sent due to a missing import statement (thanks to Ruby for not making that a fatal error). The bug was embarrassingly easy to fix once I started to look for it. The problem was that I spent Friday looking for the wrong bug because I didn’t really understand how VCR works, so I couldn’t tell correct operation from incorrect operation.

Once at another job, we thought about using Django-polymorphic. Our use case was that we had base articles with certain basic features like title, body, and description plus specialized features like lead images or tags depending on the site. Using Django-polymorphic, we could make a query against the base article class and it would return a list of article sub-class instances based on each article’s type. Best of all, it did it all “magically”: no need to specify anything extra when you run the query, the correct results will automatically be returned by the ORM!

Within days of rolling out Django-polymorphic, a bug cropped up on our sites. We couldn’t trace the issue down, but we were convinced it must be a bug in Django-polymorphic. The magical wizard had betrayed us! Finally, in desperation, we ripped Django-polymorphic out, only to find… the problem was actually unrelated.

When we patched the problem, though, we didn’t put Django-polymorphic back. Why? Because we realized that as long as we were using it but didn’t understand it, we would end up blaming all our hard to understand bugs on it, whether the magic inside of Django-polymorphic was the real cause or not. Magic was just too unpredictable. It was better to stick with something less elegant but better understood.

Any sufficiently advanced technology is indistinguishable from magic.

— Arthur C. Clarke

If you don’t know how a technology works, it seems magical to you. The flip-side of Clarke’s quote is one from Joel Spolsky:

All non-trivial abstractions, to some degree, are leaky.

Abstractions fail. Sometimes a little, sometimes a lot. There’s leakage. Things go wrong. It happens all over the place when you have abstractions.

Magic is wonderful when it works, but very few abstractions can be trusted not to break down occasionally. That’s not to argue that you can’t use abstractions. A modern computer system is nothing but a wobbling tower of poorly understood abstractions. It’s just when you aren’t aware of the abstractions your system is built on and something goes wrong, the best you can do is make like Lucy Lawless and say

A wizard did it.

Using “a wizard did it” as a plot contrivance is considered a bad way of writing fiction because the audience never knows what the stakes are. If the heroes get into trouble could a wizard just randomly save them? Or if they are about to win, could a wizard just randomly defeat them? The existence of wizards removes all narrative tension because the universe has no certainties.

Krupal Shah wrote a good summation of my thoughts on abstraction:

I don’t think that libraries and frameworks should provide the abstraction in such way that hides the details of the underlying system. Rather, they should provide the abstraction in a way that people can actually understand the underlying system. Too much abstraction becomes a barrier for most people, especially beginners to actually understand what the system does. It makes them think more about doing things rather than going deep and understanding things. That’s not a good thing. We want people to actually understand the details of stuff under the hood, not to learn the ways of using things.

People are willing to put up with magic in TV and movies as long as there seem to be rules that guide what the wizards can or cannot do. We’re in the middle of genre boom for world building fiction (which probably goes too far as it tries to fulfill some sort of psychic need for order in early twenty-first century America) in part because people like learning about system that’s can be fully understood and mastered.

Julia Evans just gave a talk at Strange Loop where she argues instead of sitting around and waiting for a wizard to do it, you can be the wizard!

I used to think to debug things, I had to be really really smart. I thought I had to stare at the code, and think really hard, and magically intuit what the bug was.

It turns out that this isn’t true at all! If you have the right tools, fixing bugs can be really easy. Sometimes with just a little more information, you can figure out what’s happening without being a genius at all.

By mastering debugging tools the let you go deeper into your stack, you can reverse engineer the magic behind whatever is happening your program. You can turn magic back into sufficiently advanced technology. You can follow the leaky abstractions back to the source pipe.

Steve Jobs said,

Everything around you that you call life was made up by people that were no smarter than you and you can change it, you can influence it, you can build your own things that other people can use.

That’s not really true, but in the world of software, it’s close enough to true. One of the real turning points in programming is when you look at some framework or magical tool and say, “I could have made that too. I just didn’t have the time or opportunity.” Once you realize that you could have made any one of the tools you use (but you didn’t because no one has that kind of time), you won’t be afraid anymore to look behind the curtain and find out who the wizard really is: