Steve Yegge on Google's Engineering Culture

The ever-voluminous Steve Yegge wrote an interesting post back in 2012 about liberal and conservative elements in programming. By "conservative", he means practices that emphasize safety and rigor; "liberal", conversely, emphasize freedom and flexibility. He then used these labels to categorize programming languages, features, development practices, and cultures.

Here's what he had to say about Google's engineering culture:

Google -- Diagnosis: Conservative. They began life as slightly liberal and have grown more conservative ever since. Google was only software-liberal in the very very early days, back when the search engine itself was written in Python. As they grew, they quickly acquired a software conservatism driven entirely by the engineers themselves.  Manifestos were written about the dangers of using multiple languages, and strict style guides were put in place to severely limit "risky" or "hard to read" language features of the few languages they did allow.  ...  In internal surveys, Google engineers commonly cite bureaucracy, churn and complexity as core obstacles to feature advancement and rapid launches.  Google has made serious attempts on several occasions to reduce this bureaucracy, but they always get pushback from -- surprise -- the engineers themselves, who have grown so staunchly conservative that they actively (and even more often, passively) resist the introduction of more flexible stacks and technologies.  ...

Having seen Google from the inside, I can say Steve nailed it. I had expected Google to be sort of an over-grown startup, a really intense place where you had to work extremely hard, but where you had a lot of freedom to do things however you wanted, as long as it worked. What I found was something quite different: a culture that was incredibly strict[1], top-down[2], and bureaucratic[3].

If Steve is right, what I expected to find at Google is actually found at Amazon and Facebook.

[1]: Check out the C++ style guide (click "Toggle all summaries" to see the document in its full form). Then consider that the code review culture routinely enforces all of it. I remember having check-ins rejected for spaces at the ends of lines, for example.

[2]: I still think I should have quit or transferred the day my lead rejected the fifth rewrite of a design document.

[3]: The most prominent manifestation of Google's bureaucratic nature is the "promo packet", an elaborate document used to apply for promotion. It is typically at least a dozen pages long, with a long series of testimonials by the applicant's bosses and peers singing his praises for the promotion committee. It's like going up for tenure or a partnership, but it's done for every step up, not just once in your career.

What was the best decade ever in computing?

If we define "decade" as ten contiguous calendar years, I think it would be hard to beat 1968-77, which brought us:

1968 structured programming ("Go To Statement Considered Harmful")
1969 UNIX
1970 PDP-11, relational algebra
1972 C
1973 Xerox Alto
1977 VAX-11/780, Apple II

A little later, we had Ethernet in 1980 and the IBM PC in 1981.

Stop White-Board Coding

It is part of my job to interview candidates for software development positions. We do this in the same way a lot of other companies do it (notably Amazon and Microsoft), using a long series of interviews where candidates answer coding questions in 45-minute slots, typically writing their answers on white-boards. This format is common enough that there are whole books devoted to passing such interviews [ref][ref].

The problem with this technique is that these interviews are a deeply artificial work environment. No one writes production code in 45-minute slots on white-boards. The time is much too short, which means the problems are much too small. Most problems would in fact fit well into second-year algorithms and data structures classes. Also, professional developers don't work on white-boards; they work with compilers and editors, or possibly IDEs.

This way of interviewing has few fans; we do it because coming up with anything better is hard. In my opinion, the actual underlying problem is that software developers don't have portfolios of work we can show. If we had actual pro-level code we could show to prospective employers, that would obviously be better than anything we could come up with on the spot. But we typically don't have it, because we typically do our work for large employers who guard their codebases closely.

So what to do? Really, the best solution depends on the level of the employee being hired.

For new grads, we should use internship programs, and hire only students who completed one and did well. What a student gets done during the three to six months of an internship is a far better signal than anything we can get out of a series of interviews. It is based on a far longer period of observation, covers a much larger body of work and (most crucially) that work was done under the company's actual working conditions.

At the other end of the scale, for rather senior developers, ten years into their careers or more, we should be hiring on the strength of the record. At this point, the issue should really not be the minutiae of coding; anyone with this much experience should be able to write sound code. The real issues are good design in the large, project management, and leadership.

Candidates this far into their careers should have strings of significant projects behind them. Interviews should consist of having them explain what problems their projects solved, how they designed solutions, and what choices they made and rejected during the design process. The goal of interviews like this is to establish that the candidates in fact did the work they claim to have done, and that they have the technical and social savvy expected of senior staff.

That leaves the hard case, candidates for fairly junior development jobs that are not entry level. Neither of the earlier solutions is really applicable; these candidates are not interested in internships, and can well have spent the early years of their careers doing really inglorious bug-fixing that doesn't showcase substantial design skills. And at this level, people really are hired to code, which some people don't do well.

This is where code review is most useful. Candidates should be expected to present code for review by several developers, and answer questions about why they wrote it the way they did.

The question is how to get access to actual code, when most of it belongs to the candidates' employers. This is a hard problem, but the burgeoning open-source movement offers a solution. Open-source developers have code they can show, because the code they work on is openly available. Accordingly, let it be known that mid-tier developers will be hired only on the basis of code that is available for review. Maintain a list of well-regarded open-source projects -- ideally ones that the company actually uses in-house -- and refer interested candidates to them. This is particularly useful since in-house developers already know their way around these projects, and can therefore judge not only the candidates' coding, but also their general behavior in the project forums.

So, to summarize:

• Hiring software developers on the basis of code scribbled on white-boards is bad.
• We should hire new grads based on internship performance.
• We should hire senior developers based of their records of design, implementation, and leadership.
• We should hire mid-tier developers based on code review, typically based on open-source code.

Jobs requiring functional programming

If you want a job as a software developer, how useful is it to know functional programming languages?

To answer this question, I went to four employment sites and searched for four functional languages: Scala, Erlang, Clojure, and Haskell. For calibration, I also searched for three mainstream programming languages: Java, C++, and SQL. Two of the employment sites were Canadian (Workopolis and Monster.ca) and two were American (Monster.com and careers.joelonsoftware.com).

Workopolis:

Java     696
C++      282
SQL     1368
Scala      3
Erlang     2
Clojure    1
Haskell    1

Monster.ca:

Java     668
C++      288
SQL     1000+
Scala      2
Erlang     0
Clojure    0
Haskell    0

Monster.com:

Java    1000+
C++     1000+
SQL     1000+
Scala     72
Erlang    36
Clojure   12
Haskell   26

careers.joelonsoftware.com:

Java     252
C++      140
SQL      231
Scala     24
Erlang    11
Clojure    9
Haskell    9

Based on these figures, there are some jobs out there that call for functional-programming expertise, but not many, particularly compared to the number calling for mainstream imperative languages. And virtually none of them are in Canada.

Valve's corporate culture

So I've checked out the Valve employee handbook that made the rounds of the internet a few months back. The corporate culture it depicts is very open and very egalitarian. 

Picking a project:

We’ve heard that other companies have people allocate a percentage of their time to self-directed projects. At Valve, that percentage is 100. Since Valve is flat, people don’t join projects because they’re told to. Instead, you’ll decide what to work on after asking yourself the right questions (more on that later). Employees vote on projects with their feet (or desk wheels). Strong projects are ones in which people can see demonstrated value; they staff up easily.

Structure:

Valve is not averse to all organizational structure—it crops up in many forms all the time, temporarily. But problems show up when hierarchy or codified divisions of labor either haven’t been created by the group’s members or when those structures persist for long periods of time. We believe those structures inevitably begin to serve their own needs rather than those of Valve’s customers.

Working hours:

While people occasionally choose to push themselves to work some extra hours at times when something big is going out the door, for the most part working overtime for extended periods indicates a fundamental failure in planning or communication. If this happens at Valve, it’s a sign that something needs to be reevaluated and corrected. If you’re looking around wondering why people aren't in “crunch mode,” the answer’s pretty simple. The thing we work hardest at is hiring good people, so we want them to stick around and have a good balance between work and family and the rest of the important stuff in life.

These are admirable principles. I would love to work Valve's way.

But I have to wonder whether the handbook depicts how things are actually done at Valve. I suspect it describes an idealized version of Valve, Valve as it ought to be. And perhaps the company sometimes achieves that ideal. Also, Valve is small; right now they have only 293 employees. I suspect things would change considerably if they were to grow by an order of magnitude. Companies have a way of growing more formal and rigid as they grow.