7 steps to building an engineering competency matrix

Every engineer deserves a clear growth path so they can understand, plan, and execute on meaningful career growth. Providing a framework for this growth (we call ours a competency matrix; it’s also known as a career ladder, or professional development ladder) is important work, and the responsibility of any organization that wants to nurture and grow its employees.

Back at the beginning of 2018, we had 32 developers and a plan to double throughout the year, we already had a competency matrix, but it was woefully outdated. It focused on our more junior levels, maxing out at a level which some developers had already reached. It was also misaligned with the skills our organization had grown to value, which meant in practice, we often ignored it. It was time for a re-design.

Building a new competency matrix was a learning process, and a lengthy one, taking about eight months to complete. Along the way we discovered things we valued, as well as what the keys steps to building a career ladder are (and which ones are wasteful). While every matrix is different, and will reflect the values of the organization that wrote it, the process of producing a succinct career ladder to guide your team is consistent.

When we published our new Engineering competency matrix in December, we received many emails from teams saying they were working on similar systems. Because of this feedback, I want to share the steps we went through, and the lessons we learned, to help teams reach a productive conclusion with much less waste, and in much shorter time, than trying to figure it out from scratch.

If you want to provide your employees and reports with a clear, agreed-upon and well-defined path for growth within your organization, then this is for you.

Image via CircleCI

Step 1: Make this someone’s top priority

In retrospect, this was the biggest factor in our lengthy redesign process. I had initially taken on this project as one of my many side projects. The only time I had to dedicate to the matrix were early mornings, late nights, and weekends. This was a passion project for me, and I loved working on it, but I was not able to give it the care it needed.

When Lena Reinhard, our new Director of Engineering, joined, she took this on as her first big project. We worked together closely after this, but the fact that she now “owned” it made it immediately obvious to me how much my limited availability had been holding this project back. If I had continued on driving this during my free time, I don’t know if we would have seen it completed in 2018.

If you are taking this project on, give it the attention it deserves by giving it to someone as their #1 job.

Step 2: Agree on what you are building

Until all stakeholders agree on what your goals are, every attempt at implementation will stall.

A competency matrix is a powerful tool to set a cultural tone and direction, so in designing your matrix, you’ll have many impactful choices to make. Each one has consequences, and you’ll only get through them if the team is aligned. One of the questions that came up repeatedly for us was: is this a codification of the status quo or is this aspirational? (It took us about halfway through the process to explicitly agree it was a blend of both.)

Another key point of agreement is: who is going to be affected by this? In our case this question hinged on whether or not our new matrix would affect our Site Reliability Engineers (we decided it would). Maybe you are building a matrix for your whole company, or maybe you are building a matrix for your frontend development team. The breadth of roles affected will greatly change the competencies you choose to codify, and the level of abstraction you need to use. So ask these questions, and get explicit agreement.

Finally, we needed to agree on the goals of the matrix. We decided the primary goal was performance evaluation and growth planning with individual engineers at CircleCI. Secondarily, we wanted to use it to influence our hiring process and communicate externally what being an engineer at various levels meant. Knowing the potential uses, and the priority of those, guided certain decisions.

Step 3: Guide by your values

This, for me, was the most fun part of this entire process. This is the part where you sit down and debate “what matters to us?” We had some help from our excellent Head of HR, David Mann, who came in with note cards detailing out ~100 behavior traits that are valuable to have as a professional. These were not engineering-specific, and ranged from Communication to Political Awareness. We also utilized other publicized competency matrices to seed ideas of what could be in the running.

We also enjoyed coming up with our own values as a team. The one that sticks out most in my mind is Economic Thinking, something the management team continually discussed as being one of the key skills that differentiates good developers from great ones. We didn’t borrow that competency from any HR cliff notes or other matrices we consulted, but from the get-go, this was a key competency we knew would be in the final cut.

Start wide, dream big! It’s better to cast a wide net up front and merge and cut later.

Once you have a list, doing a few passes to merge similar competencies is a good idea, as it will save you time later in the process. For example, we merged Pragmatism and Economic Thinking, as we realized the manifestation of these competencies resulted in the same behaviors.

Step 4: Define your levels

Before you start to actually fill out the content its good to define the x-axis of the matrix: the growth in terms of title and responsibility. At CircleCI, we already had the existing set of titles we wanted to use, (E1 / Associate Software Engineer to E6 / Principal Software Engineer). Then we explicitly agreed upon on generalized responsibilities for each title.

We broke it down into two categories, both focusing on individual contributors: E1, E2, and E3 focus on mastering the skill of software engineering and becoming a highly effective IC. E4, E5, and E6 focus on utilizing those skills to scale impact and create leverage across larger and larger groups of people. Once we had this high-level guidance, we broke it down further, assigning scope to each of those execution levels. Before we started creating content, we had the following guiding principles:

  • E1 – E3: Execution of Work
    • E1: Within Task
    • E2: Within Project
    • E3: Within Team
  • E4 – E6: Utilizing skills to scale and generate leverage
    • E4: To team
    • E5: To set of related teams
    • E6: To engineering department

Of course, these are guidelines, not rules. For example, we expect E4s, E5s, and E6s to contribute to organizational Practices and Processes at differing frequencies. Career development, like all human pursuits, is messier in practice than in theory. However, by setting clear guidelines up front, you can make the low resistance paths easy, and use your time focusing on those harder, messier bits.

Step 5: Create the content

This step is definitely the most laborious. However, if you have followed steps 1-4, it should make this step much easier for you than it was for us. I tried to tackle this step multiple times throughout the process, but without the correct framework to shape my approach, I often felt aimless and scattered. Once the framework was in place, filling out the content became much more straightforward.

Given the meatiness of this step, I’m going to break it down further into sub-steps. Following this sequencing should minimize any wasted work.

  1. Define a single level for all competencies
  2. Look for opportunities to merge competencies
  3. Fill out the rest of the levels
  4. Wordsmith

Step 5.1: Define a single level for all competencies

The first step I would strongly recommend is to define one level, such as your Senior Software Engineer, for all competencies. Rather than trying to wordsmith the nuanced differences between what it means to exhibit Delivering Feedback as an E2 vs an E3, reifying what you mean by each competency by defining a single level is a very enlightening and aligning process.

We approached this slightly differently, by defining two tiles: E3 (top level individual contributor) and E4 (first position of scale and leverage). Given our bifurcated approach to executing vs scaling, this gave us a better picture of what growth looked like through a pivotal step in a developer’s career at CircleCI.

Step 5.2: Look for opportunities to merge competencies

An amazing thing will happen while you are defining that single level for every competency: you’ll realize some of the values point at the same behaviors. This is an opportunity to merge!

After we had defined single levels, we noticed that we had two tiles that were eerily similar: Self Starting and Delivery Accountability. Although these competencies had appeared different, the behaviors we had codified were similar enough that we merged them into a single tile: Reliability and Delivery Accountability.

Early on, we made the explicit agreement that we wanted our matrix to be a collectively exhaustive definition of what growth looked like for a developer at CircleCI while also being as simple as possible. Ultimately, we ended up with 27 competencies. However, when we started 5.1 we had almost 50. Defining the behaviors for each showed us where we had opportunities to consolidate.

Step 5.3: Fill out the rest of the levels & wordsmith

Now that you know you have the behaviors and competencies you want in your matrix, it’s time to fill out the rest of the cells! This step is a lot of work, but an important step as it is the core of the matrix.

During this step would be the best time to start wordsmithing. While you are in the throes of defining the nitty gritty details of what it means to scale Economic Thinking to one team vs many is the best time to be critical about whether or not the language you are using properly conveys the intent. I would strongly advise against wordsmithing before this, but this would be a great time to revisit the tiles you defined in your first pass.

Step 5.4: Polish & consistency

I have no doubt during your run through defining each individual tile you will learn things that you want to apply to tiles you have already defined. About halfway through the process, Lena and I discovered that we used “often” and “usually” to mean the same thing at about a 50/50 rate. Since consistency was a big focus of ours, we decided on “usually” and put it in a proverbial parking lot to apply later.

I would suggest a similar approach. As you find opportunities for polish (such as replacing all occurrences of “often” with “usually”), write them down, and use that as your to-do list for a final pass. That way if you have learnings that contradict each other, you can resolve those conflicts before application, minimizing churn and ensuring greater consistency.

Step 6: Involve the people it will affect

Great, you have a competency matrix, you’re done! Actually, not quite. To us the idea that Lena and I would sit down, write up a bunch of values and behaviors, get it signed off by our VP of Engineering, and then present it to the world as final was a laughable proposition. This was going to affect our culture, our hiring, and most importantly, shape career growth for all the people we work with and whose careers we care about. We had been testing different versions and aspects of the matrix throughout the months it took to design it, but now it was time to really put it to the test. With our first pass done, we set out to garner feedback.

Lena orchestrated an excellent feedback session in the format of a focus group, made up of individual contributors of all levels, managers, and HR, that had both asynchronous and synchronous portions. There are many ways to gather feedback, such as having a diverse portion of individual contributors complete mock self-evaluations with their manager. Regardless of how you do it, the important thing is ensuring you gather feedback from the people this will affect.

The feedback we received was extremely valuable for two reasons: it surfaced good questions, which led us to tweak the matrix so it conveyed what we intended. This process also ensured we had codified the values of the organization, not just a few managers, and that the value progression was representative of how our engineers saw their career growing. This step was extremely important in generating confidence that what we had developed represented what we wanted, and would be well received.

Step 7: Ship it!

The best step of all – ship it! You have a finished product that has been wordsmithed, stress-tested, and has buy-in from respected individuals in your organization. However, even after you following a thorough process, it will never be perfect. As people start to use and test what you have created they will find slight inconsistencies and opportunities for improvement. Be ready for feedback now, and on an ongoing basis.

There are two mechanisms you should put in place now that you have your matrix: a way to gather feedback and an agreed-upon timeline for addressing it. Constantly iterating on your competency matrix would be a bad idea, because it means constantly moving the goal posts for career development at your organization. However, pretending that it is perfect and will never change is foolhardy and unrealistic.

We’ve developed a mechanism for collecting feedback (spoiler alert: it’s a Google Doc) and will be revisiting the matrix at the six-month mark to address what we’ve learned. This provides a level of stability that is important, while also allowing for iteration and evolution over time. I suggest you provide a mechanism to achieve the same goals.

If you have your own thoughts, experiences, or questions about building a competency matrix, I’d love to hear them! Feel free to reach out to me via email (justin@circleci.com) or via Twitter (@JustinC474).