Software Maintenance is Like Golf

I think software maintenance is one of the least understood concepts among engineering managers. By maintenance I mean all the small little tasks developers do to make their code nicer to work with, refactoring, testing, as well as fixing bugs. This part of the development process is difficult to manage for several reasons. For one, developers tend to be quite bad at making a case for why these activities are necessary. Maintenance is essentially a technical task so there is a mismatch in communication between decision makers and developers that is difficult to overcome. Often a developers arguments for refactoring reduce to aesthetic principals of best practices for coding that are difficult to reason about clearly but may still be impactful on the outcome of the project. The decision to spend resources for maintenance activities must involve a degree of trust in developers to spend their time wisely. And my experience tells me that often they don’t, so maybe managers are right to be a little skeptical.

While a little skepticism towards maintenance activities is healthy for managers, I believe this point of view when taken to extremes can cause managers to hold an inaccurate model of how their software is actually being developed. I once had a manager say to me that “refactoring is a bad word” and in my personal experience, tasks related to code quality have been the most difficult to pitch because this bias is so common. In fact, Stripe recently published a paper which identifies these activities as waste with the implication that if developers just wrote their code correctly the first time, we could save 85 billion dollars in lost GDP per year.

While it’s possible to be an effective manager with this simple heuristic, I think there’s a different understanding of the process of developing software that is closer to reality although maybe a bit less intuitive. The uncomfortable fact is that maintenance activities are seemingly unavoidable and sometimes refactoring really is the most impactful thing your developers can be doing for the outcome of the project. While these decisions may rarely make or break the project, having a good understanding of software maintenance is a great way to become a more effective leader and gain respect among your engineers.

The Current Metaphore

Metaphores and language shape the culture of our teams. Just like the ancient Greeks created myths to explain the chaos of the world in human terms, we do the same thing as engineers and managers. The current metaphore of maintenance is understood in the same way as home maintenance. For instance, my kitchen sink clogged up recently so I had to have a plumber come to the house to fix it and he charged me $300. It’s easy to understand this cost as a waste. My sink worked perfectly well, then I called the plumber and the end result is the same as it was before the clog: a working sink. If the plumber would have offered to rearrange the pipes to be more efficient for a cost of an extra $500, I would have respectfully declined.

Depreciation is a real phenominon for code, but it happens for a different reason than plumbing or factory equipment. The reason code requires maintenance is not because it wears out when you use it. Code is just a description of a deterministic logical process. Given the same inputs and state of your hardware, you will get the same sort of result now that you will get 20 years from now. Rather, code depreciates because people change the way they use it over time.

With the home maintenance metaphore, bugs are understood to be clear cut and well defined (we call these regressions: when something used to work and now it doesn’t), but the overwhelming majority of bugs I’ve encountered are not regressions, but rather come from the user using the software in a way that was not expected by the original designer. The user did something that seemed like it should have worked, but then the software did something else entirely. In this case, the line between a bug and a feature is not clear and it’s often not useful to make the distinction at all. For instance, if the user installs my software in an environment I didn’t do any tests for and runs into problems, is it a bug because the software is not working correctly or a feature to add support for the new environment? Whatever we call it, usually it doesn’t affect the discussion so we don’t bother with semantics. Sometimes we just tell the person we don’t want to support their use case and close the bug as wontfix. Windows users should be used to this by now.

Most “maintenance” work is actually feature development in disguise. The cause of most bugs are changes in user expectations. Your developers want to refactor the code base because they are anticipating changes in user expectations and they want to get an early start implementing the features they think you’ll need while they have the problem fresh in their mind. If you understand things this way, you should see that the home maintenance metaphore is limiting for practical decision making about maintenance activities.

The Golf Metaphore

I rather see maintenance not as a wasteful activity, but rather an important part of the development process. To me, a software project is a lot like a round of golf. The object of the game of golf is to get a ball into a hole across a field using clubs with as little effort as possible. Effort is measured by strokes, or how many times you hit the ball.

On the first stroke (the drive), you are very far away from the hole. The objective of this stroke is not necessarily to get the ball in the hole. It could happen, but getting a hole in one is really just a lucky outcome that can’t be attributed to the skill of the golfer. There are a lot of factors the golfer is not thinking about at this point like the exact speed of the wind that may alter the ball’s trajectory by several feet. The best you can hope for is to get close enough to make the rest of your shots easier.

Let’s notice some things about a good drive. First of all, the ball travels about 80% of the distance to the hole during this shot. Second, this shot costs the same as any other: one stroke. If you do a naive calculation of distance per stroke, you will come to the conclusion that your drive is by far your most efficient shot. With this data in hand, a good golf coach might tell his player your drive is your most efficient shot, so just do that every time. Worse yet, if you golf as a team and have a designated driver, you might mistakenly think this is your star player because he moves the ball the farthest. Truly this must be a 10x golfer!

But in reality you can’t drive every time (even if you could, you may never get to the hole). Your next shot requires a different set of skills and even a different set of clubs. The ball will travel much less distance during this shot, but it still counts the same as the shot before: 1 stroke. Finally, you are close enough that it’s time to putt. The putt uses the smallest club and causes the ball to move the shortest distance, but the effort required is the same as the drive.

Software maintenance work is like putting. At this point, small details matter like the contours of the earth and the length of the grass. And while this shot is not nearly as efficient as the drive, it’s the best way to play golf (as long as you don’t make the mistake of doing it too early). Some people even make a game out of just this part: minigolf and it’s pretty fun.

So if you think about a software project like this, you should see that maintenance work is just part of software development just like putting is part of golf. If your code base needs refactoring or you have bugs, it doesn’t mean that anybody made a mistake or wrote “bad code” just like a golfer who doesn’t make a hole in one isn’t a bad golfer. That’s just how golf works.

My Definition of a Hacker

The word hacker can be an charged word that means different things to different people. Usually this word is used among people outside of the software industry to refer to a computer criminal, but there are alternative definitions that predate this that are still in common use today within the industry that don’t have anything to do with criminal behavior (such as the popular forum Hacker News). In this post, I would like to explore the different uses of this word and try to come up with an alternative definition that unifies them all.

Alternative Definitions within the Software Industry

Within the software industry, the term is sometimes used to describe a talented computer programmer who enjoys solving problems with code. The common definition can be used within the industry as well, but this might be seen by some as controversial. In an influential attempt to define the term in 2001, Eric S. Raymond, a legendary self-described “hacker”, made this distinction.

There is another group of people who loudly call themselves hackers, but aren’t. These are people (mainly adolescent males) who get a kick out of breaking into computers and phreaking the phone system. Real hackers call these people ‘crackers’ and want nothing to do with them.

While I agree that the word can and should be used to describe noncriminal behavior, if a colleague calls me in the middle of the night and says to me “hackers just broke into our database”, I’m not going to spend any time correcting his usage. The word cracker seems to have fallen out of style because I’ve never heard anyone use it.

Rather, the word hacker is more often used as a term of endearment among software engineers to denote membership in a particular subculture. Between developers, a hacker is someone who enjoys solving difficult problems in a creative way. Richard Stallman, another legendary hacker, puts it like this:

Hacking means exploring the limits of what is possible, in a spirit of playful cleverness.

Hacker culture values a healthy skepticism of authority and the status quo in search of more effective and more interesting solutions to common problems. You can hear echos of the hacker ethos in the present Silicon Valley ideal of the disruption of established industries. In fact, both Stallman and Raymond were particularly disruptive to the software industry in their time by developing the Free and Open Source Licenses which now form the legal basis for how companies and individuals share common code bases with each other.

There is one more common usage of the word hack in the industry that is not as flattering. It refers to a solution to a problem that is either seen as overly complicated, fragile, or uses a low-level interface to accomplish a high-level task. This usage is somewhat related to the common definition of “a professional who is bad at something” (such as the comedian I saw last night is a hack), but in programming the word is more specific and I think it’s more related to the above two definitions as this article explains well. The word can also be used positively to describe “a clever hack” which is a surprising use of an algorithm or obscure interface to accomplish a task in an unexpected way. This is the sense of the word in the phrase “life hack”.

Putting it all Together

These definitions might seem to be incompatible with each other, but I think they are related in a simple way. Let’s review them and try to find out what they have in common.

  1. Someone who breaks into computer systems
  2. A creative and disruptive programmer
  3. Inappropriate or surprising use of low-level interfaces

I would like to propose a definition that unifies all of these usages.

A hacker is someone who creates an interface where none existed before.

[note: I think this is an original insight, but please let me know in the comments if someone has already put it like this].

Thus, hackers are people who create new interfaces. This definitely describes the pioneers of the early Internet protocols and the creators of the GNU operating system like Richard Stallman who made a Free Unix interface. These people are legendary hackers and they deserve our praise. It also describes people who break into systems and steal data. For instance, there is not an interface on your banking website that lets you take money out of someone else’s account and put it into yours. If you create this interface (hopefully using a surprising low-level approach!) then you are a hacker and you should go to jail.

Let me explain what I mean in an analogy of a house. A common saying is when God closes a door, he opens a window. Entering a house through the window is a hack in all three senses. It is 1) a way to subvert God’s security system of the house by bypassing the lock on the door to gain access, 2) a creative solution to a technical problem which disrupts our notion of home entry, and 3) the abuse of an interface to do something it was not designed to do. I think all hacks have these elements to them. While good hacks are not criminal, there is still a sort of a sense of mischief to them and a spirit of doing things you are not supposed to do. Someone who solves a problem in the way they are “supposed” to do it is definitely not a hacker after all. And of course, to create any new interface requires using a lower level interface by definition.

Don’t get me wrong, I’m not trying to compare the hacker community to criminals. These are two distinct groups of people with very different goals. But maybe these similarities are why the terms got tangled up together in the media.

Command Line vs GUI

An important topic in interface design for end users that is rarely discussed is when to design an application for the command line and when to design it as a GUI. Most computer users have a strong opinion about this. Nontechnical people often seem to hold the belief that the GUI is simply more advanced technology and command line applications are an artifact of a primitive era, while some hardcore adherents to the philosophy of Unix believe that command line applications are superior. I believe that the truth is somewhere in the middle and each paradigm of UI design can be applied appropriately to different use cases. In this blog post, I will give an outline of things to consider when choosing between the two when designing or using an application.

Technical Literacy of the Users

First consider how technically literate your users are. Not everybody who uses a computer is necessarily a computer person and that’s perfectly ok. Technical literacy proceeds much like normal literacy. When people first learn to read, they tend to start off with picture books. The pictures in a picture book give context to what is going on with the story. When words are necessary, the context of the pictures can help the reader to understand words or concepts they are not familiar with. This is a bit like the icons of a GUI which can give some visual context to a command that might not be easy to understand with words alone.

As we learn to read, we tend to prefer novels. Words are capable of expressing certain subtleties that pictures alone cannot. Some complex processes are much better modeled with words than pictures. What we lose with the visual immediacy of the GUI is made up with a greater capacity for complexity. Anyone who has put together furniture with the picture-only instructions from Ikea might know what I mean (or is that just me?).

Takeaway: always use a GUI if your users are nontechnical.

Composability and Automation

The most obvious weakness of a GUI is that they are notoriously difficult to compose and automate. By compose I mean the ability to take the output of one program and use it as the input of another program. Command line applications can be combined and assembled into scripts that can accomplish a task automatically on certain events. This isn’t possible with GUIs because they require user interaction at runtime. Command line applications are like Lego blocks that can be assembled into whatever you need at the time and rearranged when your needs change.

This has a strong influence on the interface design in either paradigm. GUI applications need to have strong and well thought out opinions on how the user will use the application. They live alone at the end of a chain of automation so they must provide everything the user needs to accomplish their task. Thus GUI applications tend to have a much larger scope than command line applications making them more difficult to develop and maintain. This isn’t all bad though because highly opinionated software tends to be easier to use, given you use it for the purpose it is designed for.

Command line applications can be designed with a limited scope to accomplish a general task. This is in line with an important tenant of UNIX design philosophy:

Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new “features”. – Doug McIlroy

This design approach is sometimes called “worse is better”. A design that has less features (“worse”) will be simpler to use and understand (“better”). Less opinionated software has the advantage of being more extensible, that is, able to be used for a purpose that the designers of the software did not anticipate.

Takeaway: prefer the command line when your application might be used as part of a larger workflow.

The Visualness of the Domain

The most obvious weakness of command line applications and text interfaces is that they are not capable of displaying raster or vector data (e.g., images or elevation data) in their most natural visual form. A full-featured web browser or image editor must always be written as a GUI.

In this situation, it’s often a good idea to provide both a command line interface and a GUI. For instance, when I’m doing image processing, I will start with GUI image editing software like Gimp and then when the task becomes more clear, I’ll switch to a command line image editing application like imagemagick which can be automated. Doing a simple task like resizing all the images in a directory is much easier to do on the command line than with a GUI image editor, but determining exactly what is the right size is easier in the GUI.

Takeaway: if your domain is visual, provide both a command line and a GUI application.

Documentation and Discoverability

GUI applications are known for being difficult to formally document. Instructions on how to do things are presented with screen shots often with little circles around the right places to click which the user must repeat themselves to get the right result. Following these instructions for especially complex tasks can start to feel like a scavenger hunt. Seemingly innocuous changes in the UI can cause the documentation to go out of date because the screenshots no longer reflect what the application looks like and updating all the screenshots is tedious. The reality is that documentation for GUIs is rarely helpful and therefore rarely used. Instead, users expect the interface itself to guide them towards their goal. GUIs are great at this because the flow of the application can be laid out visually in a way anyone can understand without reading anything.

This experience isn’t possible with a command line application. Reading (and therefore documentation) is simply required to use the software either by passing a --help flag or reading the installed manual page. However, command line applications can be documented much better because the documentation medium is the same as the command medium (text!). Often times, documentation can simply be evaluated directly by copying and pasting an example command directly from the manual page or your web browser. This makes command line applications easier to support through text based communication like email or instant messaging, because telling someone what to type is easier than telling them where to click when you can’t see their screen.

On a related note, GUIs are better at internationalization. Command line applications and their parameters are almost always in English which not everybody knows, while GUIs can provide an emersive environment in many different languages.

Takeaway: use a GUI if your users don’t like to read.

Portability

The greatest advantage of command line applications is you can run them on systems without a graphical environment. As long as you have a shell on the computer, such as with a remote shell protocol like SSH, you can run the command. This is invaluable when you need to run your application on a remote server or an embedded device. A GUI application can normally only run on a computer with a connected mouse, keyboard, and monitor. Remote desktop protocols exist, but they are heavy on resources and much less reliable than SSH. If your workflow primarily consists of command line applications, it’s possible to work seamlessly on many machines at once which is a big boost to productivity. This is the reason why I prefer to use a lightweight text editor like Vim instead of an IDE.

Takeaway: use the command line if you need to run on servers and embedded devices.

Conclusion

This all might seem obvious, but you’d be surprised at how often I see people choosing the wrong tool for the job. Using both command line applications and GUI applications are an essential part of being a computer power user and knowing when to use each one is an important skill for computer programmers who want to work as efficiently as possible.