nemil

Better Software Engineering Debates
“Ruby isn’t fast enough.”
“I can’t believe anyone in 2016 is using anything other than the Java Virtual Machine (Scala, Clojure) to build a webapp.”
“Mongo is the best database; SQL is terrible and SQL databases aren’t scalable. Any modern engineer has to know Mongo.”
“Golang sucks (it doesn’t have generics); I can’t believe anyone would use it.”

When I first became a software engineer, I was surprised by the term “religious war” to refer to the angry technical debates that could be had. I wanted to understand what made certain engineering arguments a “religious war” rather than something more consistent with our engineering roots (namely, empiricism and the scientific method). I also became alarmed at the poor engineering choices made after weak technical debates with strong emotions.

Many engineering debates can be emotional or reductionist, but have little to do with engineering benefits, even though they’re couched in those terms[0]. By examining these debates and offering suggestions, I hope to help engineers early in their careers navigate the conversations ahead, as well as those who interact with software engineers.

My key points are:

  1. Be effective at identifying factors that can reduce the quality of software engineering debates (someone with skin the game, using hype/content marketing as engineering inputs, reducing arguments to good vs. bad, favoring vanity metrics)
  2. Assess yourself, especially your predilection for your chosen tools
  3. Assess content and people that you'll use to make engineering decisions, and understand any potential issues in the source
  4. Encourage thoughtful software debates, by discussing tradeoffs and highlighting factors that are reducing the quality of the debate

Let’s start by looking at a few factors that diminish the quality of software engineering debates.

“Skin in the Game”

The most prevalent reason for an emotional, but ineffective engineering debate is when parties have “skin in the game” – leading to winners and losers[1]. One example is a consulting shop that knows a single language/framework and is extolling the engineering virtues of their choice compared to all others. Another is a group of engineers who have to make a tradeoff that will affect them differently, especially when related to job or promotion prospects. While I can’t speak to the specifics of Uber’s internal debate about Node vs. Python, I know companies where such a debate would be far divorced from the actual technical merits – even when had in those terms (X is faster, Y is scalable, Z requires less maintenance).

“Skin in the game” applies more than just when job or revenue prospects on the line. A common example is our desire to not spend time learning something new – or focus on technology we want to learn, even if it’s a bad choice for the team. You may be working with a team that knows a technology well and still strongly advocate for a tool you know instead. This is due to the fact that it’s easier for you. Or you may be thinking about your career prospects and so advocate for a technology that will help you, even if it is at odds with what is “right” for your team. These technology choices may then live on in your stack, increasing complexity without providing much benefit.

Hype and Marketing

Hype and content marketing are an especially powerful ally for those with “skin in the game,” where motivated parties need broader user/developer uptake for success. A new operating system, social network, open source frontend framework, or language requires users and developers before it becomes competitive and valuable – and some of its proponents try to maximize the size and growth of users/contributors however possible – especially partially obscured marketing. Network effects are critical for these ideas, and so can lead to strong debates not tied to actual engineering.

This can be particularly problematic on engineering topics, where deep subject matter expertise is a prerequisite – allowing someone with even a rudimentary understanding to control the discussion (enough to confidently opine and be dangerous, but not enough to truly understand). And even thoughtful engineers may not have much choice about whether to learn a certain technology, if parties with “skin in the game” are able to reach an insurmountable critical mass.

Furthermore, extreme statements and buzzwords get wider coverage and attract supporters[2], so there’s little value to nuance when having public debates. The motives don’t even have to be so malevolent, as an open source project or language may be competing for contributors against other open source projects and selling the art of the possible (fanciful?). The content marketing[3] of these companies/open source projects (talks, blog posts, media coverage, college classes led by those who personally benefit) then sneaks into engineering debates, not thoughtful, accurate points made by dispassionate engineers.

Hype[4] can be successful because some outsource their thinking – they may not have the technical expertise or time or defer to those with experience, without realizing that some proselytizers have skin in the game. As a result, they are swayed by weak arguments in one-sided media coverage or posts upvoted by a passionate minority. Some don’t have the skills to dig deep into claims, and so will echo what they hear evangelized, rather than realizing how the parties making the claim have much to gain, and your team much to lose.

Good versus Bad

Another consistent flaw in our engineering debates is that those arguing will have different use cases in mind, but then advocate that the technology is “good” or “bad” for all based on how it meets their own use case.

Consider the always-fun text editor flame wars between Vim and Emacs – one can reduce part of the debate[6] down to a single question: how fully featured do you want your text editor out of the box? If you’re a DevOps engineer who spins up new environments constantly you’ll have different ideas about the best choice compared to a full stack engineer working in a single development environment that can be painstakingly setup. For the DevOps engineer, a light text editor that is installed by default is likely ideal, while for a developer with a single computer, a full-featured IDE (auto-completion, plug-ins, custom keys) is worth the high setup cost. On the other hand, the debate will focus on whether one is “good” versus “bad”, not which is the right tool for a given use case.[7]

(Even here, there is “skin in the game” on two levels: Those who have spent years fine-tuning their editor may not want to switch regardless of benefits, and a Vim or Emacs contributor might benefit from the demise of the other, leading to an influx of contributors and/or users – and making it harder for competitors to build alternatives.)

This “good vs. bad” thinking can be especially problematic when hype leads your team to choose technologies/techniques you shouldn’t be using. For example, the recent excitement around microservices (in some ways, a rebranding of practices used before) led some to refactor their codebases, even though in early companies there are huge productivity benefits to monoliths. Some teams would then focus on microservices at the start – rather than product/market fit in a monolith – their engineering schedules a victim to the hype around containerization.

And while I’ve focused on a single way we can be overly reductionist in our debates, you can think of others (e.g., old vs. new).

Favoring Vanity Features

Performance, scalability and page load size[7a] are heavily emphasized in engineering debates when they may not be critical. One hallmark of great engineers is the ability to make tradeoffs given the problem at hand.[8] As such, by choosing a highly “performant” language or minimizing page size – we’re making tradeoffs in other areas like the speed of development or the developer ecosystem or maintainability. At times, I think about our focus on these metrics similar to the way an automotive enthusiast might think about acceleration time or top speed – a fun metric that appeals to a part of us, but a meaningless metric for many important decisions. And woe to the truck manufacturer that hires an engineer who applies this mindset to their work.

Taking this a step further, our debates can focus on optimizing technical aspects without taking account of the other tradeoffs – even when the best decisions requires a global maxima, not a local technical maxima. For example, a language choice for a company/project should be made not just on the technical details (performance, scalability, memory footprint, etc.) but a myriad of other tradeoffs such as library ecosystem, size of recruitable developers, skillset of your development team, future prospects, maintenance challenges – but the debate about the “right language” is often focused solely on the technical plane. So many engineers are good about global vs. local optimization when designing their own systems or tools – but forget this when thinking about other questions that would also benefit from a global maxima.

Part of this is due to certain biases when we communicate. We’ll know our side or technology better and don’t know the other side well. We’ll then caricature the other side with the most extreme voices or failings of that side – while solely looking at the best of our side (this applies to other debates humans have, like those of the political variety). And an attack on our tools seems like an attack on us, leading to defensiveness, which is an inoculation against learning.

Having Better Engineering Debates

Engineering debates matter. In startups, having bad or angry technical debates can obscure the actual goal of the team — making something people want — and in any company they can mean the difference between success and failure.

1. Assess Yourself

It begins with assessing yourself. A number of debates you’ll have with others are due to tools you personally favor, and much will come from divorcing yourself from this predilection. One valuable exercise is to think through the number of frameworks, libraries, and even languages that you’d be willing to get into angry debates about – and reflect on how many are still around or popular even a few years later. For junior engineers, look back on a few HN front pages from 3-5 years ago, so you can realize how quaint and meaningless certain debates feel. It’ll be natural to favor and advocate for your own tools, but also spend the time to see the other side (an old debate aphorism goes: “make your opponent’s argument better than they can”). Reflect on which use cases your favored tools would actually be poor choices for – and what innovations or insights the other side has brought to the table.

Be thoughtful about when you evangelize, and don’t do it without understanding deeply the use case of who you’re advising, not just your own use case (this applies not just in conversation, but in your blog posts and comments). Be careful whenever you’re defensive or angry, as it’s likely that you’ll dismiss arguments that you might otherwise find compelling [9]. Know what you know, but just as crucially, know what you don’t know.

2. Assess External Inputs

Next, assess the external inputs into your engineering decisions. Start with assessing the engineers you interact with. Whenever you’re having a technical debate with another party, determine if any personal or parochial incentives could guide their beliefs (personal ease of use, financial/career motives, desire to build a developer ecosystem for their product, self-branding). Assess what evidence they’re using to make decisions, and see if there may be issues with these sources. If another party is evangelizing their solutions for your needs, ask yourself if they truly understand your use case, have deep content expertise, and are highlighting the tradeoffs. Be fearful if no amount of contradictory evidence would convince them.

You also have to assess the engineering content you use for decision-making. Beware of using marketing content to make engineering decisions. Think about how easy it would be for a motivated party to get a lot of upvotes on HN and high coverage – or how you might convince a group of junior engineers about something that wasn’t true. And remember that great content marketing is about obscuring the underlying intentions of the writer (security engineers talk about Sybil attacks, and a well executed, but obfuscated marketing strategy has some parallels).

If you know nothing about a certain engineering topic, spend time listing out the various claims and trying to learn by proving to yourself that the claims are in fact accurate. Always explore what tradeoffs the solution is making, including non-technical tradeoffs (which are often neglected). This may be hard, but may be important for the most critical decisions. Alternatively, develop a community of engineers around you that you truly respect, are thoughtful, and will test claims with primary research (not just reading or repeating secondary sources). Beware new terms – and acronyms – to describe old solutions, as they’re a powerful way to get coverage, sharing, and build excitement – but may not be the right choice for your team.

For learning and personal growth, favor teams that express these values, rather than content created for marketing purposes [11]. If you do use media to make decisions about engineering, you have to be able to tune out hype and good vs. bad arguments. You may also need to make adjustments when using one community (say open source software discussions) for other purposes (like startup engineering decisions).

3. Encourage Thoughtful Software Debates

Finally, you need to make adjustments to how you communicate – both in the physical world and the virtual (social networks, blog posts). Whenever getting into a heated technical discussion, ask questions rather than calling someone out directly. These questions can help you understand all the assumptions they are making, and help you lead them to a better answer (if you definitely know they’re wrong), without triggering defensiveness. Realize that even though someone may make a literal statement, often they’re guided by a point they haven’t expressed. Practice communication with nuance, if your goal is to convince thoughtful people – and encourage debates about tradeoffs instead of good or bad. Refocus conversations on the common goal of the team or group. Spend time evangelizing better engineering debates – not only languages/frameworks/tools.

A broader point is this: you need to be able to identify when a debate had in engineering terms but isn’t a real debate about engineering tradeoffs relevant to your team. There are hard engineering decisions aplenty.

Like most of what I write, I encourage debates on Hacker News (here) – and relevant points may be reflected in the essay above over time. Originally published December 2016.

feedback? drop a note to nemil at this domain
All opinions expressed are solely my own. Thanks to the following for feedback and warm-spirited debates:
  • Alain Chuard
  • Josh Cincinnati
  • Ben Lerner
  • Michael Mokrysz
  • Adam Reis
  • Deyan Vitanov
  • Tony Xiao
  • David Yang, Full Stack Academy
Footnotes
[0] In the future, I’ll address the other side of engineering “religious wars”, real engineering debates that evoke so much emotion that they make it hard to truly understand the engineering tradeoffs.
[1] This is a key reason politics exist in organizations
[2] This mirrors the benefits of extreme statements for politicians in primaries
[3] In some cases, good content marketing is just a buttoned up version of trolling, with the goal of maximizing engagement and coverage.
[4] To call out one company specifically, I was amazed how effective Mongo’s marketing machine was in its early years. On the other hand, it was terrifying to me to use its original database technology in companies with even some basic database needs (Mongo has gotten better, so the same critique doesn’t apply today). The NoSQL movement had benefits to some, and yet this wave and the accompanying hype allowed Mongo to sell their tool to organizations that shouldn’t have been using it, with the promise of easy, infinite scale. And a Mongo engineer would be the one to coin the term “MEAN stack” which sparked an overabundance of content marketing. Junior engineers I would help out in the months afterward felt that Mongo knowledge was a critical part of getting a job in the tech industry – versus general database knowledge.
[6] To reveal my own preferences, I’m a heavy user of a deeply customized Vim environment, though Emacs was my primary choice in college. And for my purposes here, I’m not going to get into modal editing, Evil mode, or other differences.
[7] In another vein, I had a discussion with one engineer who argued that “Ruby isn’t fast enough”. After I asked a question or two, he explained that he had come into his company after they were already successful – and his job was refactoring computationally heavy code into Go. Given his needs, Go was a great choice – and yet woe to the junior engineer or prospective CRUD app developer parsing his original comment as “Ruby is bad.”
[7a] I know this will be a controversial statement, and my point is not to dismiss these metrics totally – but suggest we optimize for them when they truly matter, and don’t cause undue tradeoffs. For example, scalability is generally (but not always) overrated in startups. If you’re at a large tech company with millions of users, this obviously doesn’t apply.
[8] Prospective engineers learn the Project Management Triangle: the idea that we are always making tradeoffs between faster, better, cheaper.
[9] Garry Tan, a partner at Initialized Capital and former partner at YCombinator, introduced me to the idea of “strong ideas, weakly held” which has always served me well in making decisions quickly, but being able to backtrack when warranted, without being defensive.
[10] This applies especially to people in areas where the tech talent is smaller and so are even more dependent on faraway sources that contain a substantial amount of marketing