photo of me

Lindsay Holmwood is an engineering manager living in the Australian Blue Mountains. He is the creator of Visage, Flapjack & cucumber-nagios, and organises the Sydney DevOps Meetup.

Why do you want to lead people?

Understanding your motivations for a career change into management is vitally important to understanding what kind of manager you want to be.

When I made the transition into management, I didn't have a clear idea of what my motivations were. I had vague feelings of wanting to explore the challenges of managing people. I also wanted to test myself and see if I could do as good a job as role models throughout my career.

But all of this was vague, unquantifiable feelings that took a while to get a handle on. Understanding, questioning, and clarifying my motivations was something I put a lot of thought into in the first year of my career change.

People within your teams will spend much more time than you realise looking at and analysing what you are doing, and they will pick up on what your motivations are, and where your priorities lie.

They will mimic these behaviours and motivations, both positive and negative. You are a signalling mechanism to the team about what's important and what's not.

This is a huge challenge for people making the career change! You're still working all this shit out, and you've got the ever gazing eye of your team examining and dissecting all of your actions.

These are some of the motivations I've picked up on in myself and others when trying to understand what drew me to the management career change.

Money

It is undeniable that there is a pay bump when moving to management. In most organisations, the pay ceiling is much higher in management than in engineering.

Many engineers who rise through the ranks get to a point where the only way they will earn more is if they switch from engineering to management, so that becomes the primary motivation.

The pay is higher for a good reason though - it's actually difficult to do the job well! Management looks easy from the outside, but it's difficult on the inside. Again, our friends Dunning and Kruger posit that for a given skill, incompetent people will:

  • tend to overestimate their own level of skill
  • fail to recognize genuine skill in others
  • fail to recognize the extremity of their inadequacy
  • recognize and acknowledge their own previous lack of skill, if they are exposed to training for that skill

Poor decisions are obvious and easy to criticise. Because we spend a lot of time looking at those in our organisation above us, we're finely attuned to mistakes and inadequacies, and tend to glass over the good things they do.

Understanding what about those decisions and behaviours makes sense to the people making them is difficult but vital to effectively working with others, regardless of whether you're in management, engineering, sales, finance, or operations.

More often than not, there are good reasons behind bad decisions. We are all locally rational.

The pay bump has strings attached - you're going to be making plenty of decisions, both good and bad, and wearing the consequences of them.

You are being paid to be empathetic - to understand how people are feeling, how implementing change will affect people, how to keep them motivated and working towards the big picture goal. None of these tasks are simple!

If you're primarily motivated to move into management by better pay, then you need to seriously consider how that motivation will affect the people that report to you, how mimicry of those motivations and behaviours by people in your team flow on other teams you work with, and what you need to do to meet the commitments you have to your team.

Will you be doing the bare minimum to collect your paycheck? What’s stopping you from becoming an example of the Peter Principle? What skills do you need to develop to meet your people's needs and expectations?

The hard problems in tech are not technology, they're people. That is why management pays more.

Influence

Being in management grants you power and influence in your organisation to build and run things as you see fit.

This is often a key motivation for people who want to transition from engineering to management - they have a clarity of vision and they want the power to mandate how things should be built, and implement that vision.

The motivation is always rooted in good intentions (“things could be so much more efficient if everyone just listened and did what I said”), and often results in a industrialist approach to managing people - “manager smart, worker stupid”.

The influence trap

Your influence can be wielded as a lever (leadership) or as a vise (management). Levers are useful at moving heavy objects but lack precision. Vises are very precise but a weight too heavy will slip from them.

Vises are an alluring way for first time managers to work. The vise management style is prescriptive, centrally co-ordinated, command and control. And if you watch carefully you'll soon realise it limits the potential of the team.

Prescriptive, vise-like management assumes you are the smartest person in the room, and know best how things should be done.

It doesn't multiply the teams effectiveness. The point of being a manager is to be a lever that multiplies the effectiveness of the team - to synthesise different and conflicting ideas to come to decisions and solutions nobody could have anticipated or come up by themselves. This is near impossible if you solely wield your influence as a vise.

Studies show people's problem individual performance lifts after being exposed to teamwork situations and training.

Prescriptive management increases the gap between Work As Imagined vs Work As Done. While conceptually you may have a great idea about how to solve a problem or operate a system daily, the people implementing your plans always discover gaps between the concept and implementation. Over time these gaps become larger, to the point you have a distorted view of how work is being done compared to how it's actually being carried out.

You optimise the effectiveness of the system by having tight feedback loops, open communication channels where people are rewarded for providing both negative and positive feedback about the design and operation of the system. As a manager, this means you need to be actively engaging with the people in your team - finding out what they think and feel about the work.

Finally, prescriptive management is an empirically bad way of retaining creative talent. Constant overruling and minimisation of feedback is a great way to piss people off. If you hire creative, intelligent, capable people and keep them locked in a box, they're going to break out.

Multiplying, trust, and happiness

Maybe you are the smartest person in the room, but others will bring knowledge and experience to the table you simply don't have.

You get the best out of the team by creating a safe space for people to put forward ideas, argue them without recriminations, and build consensus.

The goal for people leading high performing teams should be to have the output of the team be greater than the sum of the individual efforts of people in the team.

Your status as a manager grants you power within your organisation. That power must be wielded responsibly. You won't know if you're wielding that power responsibly in the first 12 months of the career change, at best.

You must constantly assess whether the decisions you're making are the best for the people who report to you. It's a constant tightrope act to balance the needs of your people over the needs of the business.

It's easy to pass the policy buck and say “I’m just following orders” when implementing unpopular changes, but you do have a responsibility to identify and push back on change that negatively affects people before you roll it out, and minimise the unavoidable negative effects of that change.

It does not take long for things to come apart when you take your eye off the ball and stop looking out for the team. Trust is hard to build, and easy to lose. People spend a lot of time looking at you and analysing your behaviour. They will notice much earlier than you realise when you take your eye off the ball.

It takes at least 5 positive interactions to start re-establishing trust after you've breached it.

Being in a management position grants you the power to shape how people within your organisation do their work. This means you have a direct influence over their happiness and wellbeing. Blindly implementing policy and not empathising with the people in your team can cause irreparable damage and create emotional scar tissue that will stay with people for years, if not decades.

Your power must be wielded responsibly. Do not fuck this up. When you do (and don't worry, you will, we all have), own your mistakes, apologise, and rebuild the trust.

Personal development / Career change

Personal development is a pretty good motivation for a career change to management! You want to challenge yourself to do a better job than those before and around you.

A huge personal motivation for me when moving into management was to treat others better than I had been treated throughout my career until that point.

Working in environments where the happiness of people was not the primary concern of those in charge is not a fun experience. Shared negative and stressful experiences helped me form close bonds and develop a camaraderie with the people I worked with. I couldn't say the same about the people I worked for.

Those relationships are something I value, but I wouldn't want anyone else to have to go through what we did just to obtain that sort of relationship.

The challenge for me was clear: was it possible to develop that camaraderie within the team I lead through purely positive experiences?

Looking back at how particular decisions and behaviours I experienced affected me and other people in the teams I worked in in these stressful environments, there were some obvious things that I could improve on.

There were other decisions I considered to be poor at the time, but after finding myself in similar positions I made similar choices.

I failed fairly terribly at the transition during the first 12 months of my career change. Someone in my team described my management style as “absent father”. That really put into perspective that my priorities were misplaced, and I needed to focus on the team and not my own individual performance.

My first experience working in tech was overwhelmingly positive. The working environment and management I experienced on a daily basis in the first 3 years of working in tech is the experience I aspire to create for people in the teams I lead every day.

The times I had a “good boss” are some of my best memories in my career. I was focused on the work, consistently delivered things I was excited about, and rarely worried about troubles elsewhere in the business (and it turned out there were a lot of them).

The enduring attitude from that time is the feeling of working with, not for my manager. We worked as a team to solve problems together, not as individuals off doing our own thing. That's the feeling I want to create in the teams I lead.


Understanding what motivates your career change is not an easy task.

At the end of the first year of my career change, my motivations lay somewhere between influence and personal development.

These motivations have morphed over time. Today, my focus is the happiness of the people I work with.

You need to undertake a constant process of self-reflection and a space to develop your understanding of your motivations. It’s important you create the time and space to do this!

The simplest trap to fall into in your first year is to be focused on the daily grind, the tactical details, and not think about the bigger picture.

This is something that affects experienced and novice managers alike, and it’s important to establish good personal habits early on so you have time to reflect on what motivates you, and what sort of leader you’re going to be.

Read more...

It's not a promotion - it's a career change

The biggest misconception engineers have when thinking about moving into management is they think it's a promotion.

Management is not a promotion. It is a career change.

If you want to do your leadership job effectively, you will be exercising a vastly different set of skills on a daily basis to what you are exercising as an engineer. Skills you likely haven't developed and are unaware of.

Your job is not to be an engineer. Your job is not to be a manager. Your job is to be a multiplier.

You exist to remove roadblocks and eliminate interruptions for the people you work with.

You exist to listen to people (not just hear them!), to build relationships and trust, to deliver bad news, to resolve conflict in a just way.

You exist to think about the bigger picture, ask provoking and sometimes difficult questions, and relate the big picture back to something meaningful, tangible, and actionable to the team.

You exist to advocate for the team, to promote the group and individual achievements, to gaze into unconstructive criticism and see underlying motivations, and sometimes even give up control and make sacrifices you are uncomfortable or disagree with.

You exist to make systemic improvements with the help of the people you work with.

Does this sound like engineering work?

The truth of the matter is this: you are woefully unprepared for a career in management, and you are unaware of how badly unprepared you are.

There are two main contributing factors that have put you in this position:

  • The Dunning-Kruger effect
  • Systemic undervaluation of non-technical skills in tech

Systemic undervaluation of non-technical skills

Technical skills are emphasised above all in tech. It is part of our mythology.

Technical skill is the dominant currency within our industry. It is highly valued and sought after. If you haven't read all the posts on the Hacker News front page today, or you're not running the latest releases of all your software, or you haven't recently pulled all-nighter coding sessions to ship that killer feature, you're falling behind bro.

Naturally, for an industry so unhealthily focused on technical skills, they tend to be the deciding factor for hiring people.

Non-technical skills that are lacking, like teamwork, conflict resolution, listening, and co-ordination, are often overlooked and excused away in engineering circles. They are seen as being of lesser importance than technical skills, and organisations frequently compensate for, minimise the effects of, and downplay the importance of these skills.

If you really want to see where our industry places value, just think about the terms "hard" and "soft" we use to describe and differentiate between the two groups of skills. What sort of connotations do each of those words have, and what implicit biases do they feed into and trigger?

If you're an engineer thinking about going into management, you are a product of this culture.

There are a handful of organisations that create cultural incentives to develop these non-technical skills in their engineers, but these organisations are, by and large, unicorns.

And if you want to lead people, you're in for a rude shock if you haven't developed those non-technical skills.

Because guess what - you can't lead people in the same way you write code or manage machines. If you could, management would have been automated a long time ago.

The Dunning-Kruger effect

The identification of the Dunning-Kruger effect is one of the most interesting development of modern psychology, and one of the most revelatory insights available to our industry.

In 1999 David Dunning and Justin Kruger started publishing the results of experiments on the ability of people to self-assess competence:

Dunning and Kruger proposed that, for a given skill, incompetent people will:

  • tend to overestimate their own level of skill
  • fail to recognize genuine skill in others
  • fail to recognize the extremity of their inadequacy
  • recognize and acknowledge their own previous lack of skill, if they are exposed to training for that skill

If you've had a career in tech without any leadership responsibilities, you've likely had thoughts like:

  • "Managing people can't be that hard."
  • "My boss has no idea what they are doing."
  • "I could do a better job than them."

Congratulations! You've been partaking in the Dunning-Kruger effect.

The bad news: Dunning-Kruger is exacerbated by the systemic devaluation of non-technical skills within tech.

The good news: soon after going into leadership, the scope of your lack of skill, and unawareness of your lack of skill, will become plain for you to see.

Also, everyone else around you will see it.

Multiplied impact

This is the heart of the matter: by being elevated into a position of leadership, you are being granted a responsibility over people's happiness and wellbeing.

Mistakes made due to lack of skill and awareness can cause people irreparable damage and create emotional scar tissue that will stay with people for years, if not decades.

Conversely, by developing skills and helping your team row in the same direction, you can also create positive experiences that will last with people their entire careers.

The people in your team will spend a lot of time looking up at you - far more time than what you realise. Everything you do will be analysed and disected, sometime fairly, sometimes not.

If you're not willing to push yourself, develop the skills, and fully embrace the career change, maybe you should stay on the engineering career development track.

But it's not all doom and gloom.

By striving to be a multiplier, the effects of the hard work you and the team put in can be far greater than what you can achieve individually.

You only reap the benefits of this if you shift your measure of job satisfaction from your own performance to the group's.

"Real work"

Many engineers who change into management feel disheartened because they're not getting as much "real work" done.

If you dig deeper, "real work" is always linked to their own individual performance. Of course you're not going to perform to the same level as an engineer - you're working towards the same goals, but you are each working on fundamentally different tasks to get there!

Focusing on your own skills and performance can be a tough loop to break out of - individual achievement is bound up in the same mythology as technical skills - it's something highly prized and disproportionately incentivised in much of our culture.

If you've decided to undertake this career change, it's important to treat your lack of skill as a learning opportunity, develop a hunger for learning more and developing your skills, routinely reflect on your experiences and compare yourself to your cohort.

None of these things are easy - I struggled with feelings of inadequacy in meeting the obligations of my job for the first 3 years of being in a leadership position. Once I worked out that I was tying job satisfaction to engineering performance, it was a long and hard struggle to re-link my definition of success to group performance.

If everything you've read here hasn't scared you, and you've committed to the change to management, there are three key things you can start doing to start skilling up:

  1. Do professional training.
  2. Get mentors.
  3. Educate yourself.

Training

Tech has a bias against professional training that doesn't come from universities. Engineering organisations tend to value on-the-job experience over training and certification. A big part of that comes from a lot of technical training outside of universities being a little bit shit.

Our experience of bad training in the technical domain doesn't apply to management - there is plenty of quality short course management training available, that other industries have been financing the development of the last couple of decades.

In Australia, AIM provide several courses ranging from introductory to advanced management and leadership development.

Do your research, ask around, find what people would recommend, then make the case for work to pay for it.

Mentors

Find other people in your organisation you can talk to about the challenges you are facing developing your non-technical skills. This person doesn't necessarily need to be your boss - in fact diversifying your mentors is important for developing skills to entertain multiple perspectives on the same situation.

If you're lucky, your organisation assigns new managers a buddy to act as a mentor, but professional development maturity for management skills varys widely across organisations.

If you don't have anyone in your organisation to act as a mentor or buddy, then seek out old bosses and see if they'd be willing to chat for half an hour every few weeks.

I have semi-regular breakfast catchups with a former boss from very early on in my career that are always a breath of fresh air - to the point where my wife actively encourages me to catch up because of how less stressed I am afterwards.

Another option is to find other people in your organisation also going through the same transition from engineer to manager as you. You won't have all the answers, but developing a safe space to bounce ideas around and talk about problems you're struggling with is a useful tool.

Self-education

I spend a lot of time reading and sharing articles on management and leadership - far more time than I spend on any technical content.

At the very beginning of your journey it's difficult to identify what is good and what is bad, what is gold and what is fluff. I have read a lot of crappy advice, but four years into the journey my barometer for advice is becoming more accurate.

Also, be careful of only reading things that re-inforce your existing biases and leadership knowledge. If there's a particular article I disagree with, I'll often spend a 5 minutes jotting a brief critique. I'll either get better at articulating to others what about that idea is flawed, or my perspective will become more nuanced.

It's also pertinent to note how the article made you feel, and reflect for a moment on what about the article made you to feel that way.

If you're scratching your head for where to start, I recommend Bob's Sutton "The No Asshole Rule", then "Good Boss, Bad Boss". Sutton's work is rooted in evidence based management (he's not talking out of his arse - he's been to literally thousands of companies and observed how they work), but writes in an engaging and entertaining way.


Almost four years into my career change, I can say that it's been worth it. It has not been easy. I have made plenty of mistakes, have prioritised incorrectly, and hurt people accidentally.

But so has everyone else. Nobody else has this nailed. Even the best managers are constantly learning, adapting, improving.

Think about it this way: you're going to accumulate leadership skills faster than people who have made the change because you're starting with nothing. The difference is nuance and tact that comes from experience, something you can develop by sticking with your new career.

This will only happen when you fully commit to your new career, and you change your definition for success to meet your new responsibilities as a manager.

Read more...

Applying cardiac alarm management techniques to your on-call

If alarms are more often false than true, a culture emerges on the unit in that staff may delay response to alarms, especially when staff are engaged in other patient care activities, and more important critical alarms may be missed.

One of the most difficult challenges we face in the operations field right now is "alert fatigue". Alert fatigue is a term the tech industry has borrowed from a similar term used in the medical industry, "alarm fatigue" - a phenomenon of people being so desensitised to the alarm noise from monitors that they fail to notice or react in time.

In an on-call scenario, I posit two main factors contribute to alert fatigue:

  • The accuracy of the alert.
  • The volume of alerts received by the operator.

Alert fatigue can manifest itself in many ways:

  • Operators delaying a response to an alert they've seen before because "it'll clear itself".
  • Impaired reasoning and creeping bias, due to physical or mental fatigue.
  • Poor decision making during incidents, due to an overload of alerts.

Earlier this year a story popped up about a Boston hospital that silenced alarms to improve the standard of care. It sounded counter-intuitive, but in the context of the alert fatigue problems we're facing, I wanted to get a better understanding of what they actually did, and how we could potentially apply it to our domain.

Read more...

Rethinking monitoring post-Monitorama PDX

The two key take home messages from Monitorama PDX are this:

  • We are mistakenly developing monitoring tools for ops people, not the developers who need them most.
  • Our over-reliance on strip charts as a method for visualising numerical data is hurting ops as a craft.

Death to strip charts

Two years ago when I received my hard copy of William S. Cleveland's The Elements of Graphing Data, I eagerly opened it and scoured its pages for content on how to better visualise time series data. There were a few interesting methods to improve the visual perception of data in strip charts (banking to 45˚, limiting the colour palette), but to my disappointment there were no more than ~30 pages in the 297 page tome that addressed visualising time series data.

Read more...

Flapjack, heartbeating, and one off events

Flapjack assumes a constant stream of events from upstream event producers, and this is fundamental to Flapjack's design.

a beating heart

Flapjack asks a fundamentally different question to other notification and alerting systems: "How long has a check been failing for?". Flapjack cares about the elapsed time, not the number of observed failures.

Alerting systems that depend on counting the number of observed failures to decide whether to send an alert suffer problems when the observation interval is variable.

Take this scenario with a LAMP stack running in a large cluster:

Read more...

Data driven alerting with Flapjack + Puppet + Hiera

On Monday I gave a talk at Puppet Camp Sydney 2014 about managing Flapjack data (specifically: contacts, notification rules) with Puppet + Hiera.

There was a live demo of some new Puppet types I've written to manage the data within Flapjack. This is incredibly useful if you want to configure how your on-call are notified from within Puppet.

Video:

Slides:

The code is a little rough around the edges, but you can try it out at in the puppet-type branch on vagrant-flapjack.

Read more...

The questions that should have been asked after the RMS outage

Routine error caused NSW Roads and Maritime outage

The NSW Roads and Maritime Services' driver and vehicle registration service suffered a full-day outage on Wednesday due to human error during a routine exercise, an initial review has determined.

Insiders told ITnews that the outage, which affected services for most of Wednesday, was triggered by an error made by a database administrator employed by outsourced IT supplier, Fujitsu.

The technician had made changes to what was assumed to be the test environment for RMS' Driver and Vehicle system (DRIVES), which processes some 25 million transactions a year, only to discover the changes were being made to a production system, iTnews was told.

There is a lot to digest here, so let's start our analysis with two simple and innocuous words in the opening paragraph: "routine exercise".

Read more...

The How and Why of Flapjack

In October @rodjek asked on Twitter:

"I've got a working Nagios (and maybe Pagerduty) setup at the moment. Why and how should I go about integrating Flapjack?"

Flapjack will be immediately useful to you if:

  • You want to identify failures faster by rolling up your alerts across multiple monitoring systems.
  • You monitor infrastructures that have multiple teams responsible for keeping them up.
  • Your monitoring infrastructure is multitenant, and each customer has a bespoke alerting strategy.
  • You want to dip your toe in the water and try alternative check execution engines like Sensu, Icinga, or cron in parallel to Nagios.

Read more...

CLI testing with RSpec and Cucumber-less Aruba

At Bulletproof, we are increasingly finding home brew systems tools are critical to delivering services to customers.

These tools are generally wrapping a collection of libraries and other general Open Source tools to solve specific business problems, like automating a service delivery pipeline.

Traditionally these systems tools tend to lack good tests (or simply any tests) for a number of reasons:

  • The tools are quick and dirty
  • The tools model business processes that are often in flux
  • The tools are written by systems administrators

Sysadmins don't necessarily have a strong background in software development. They are likely proficient in Bash, and have hacked a little Python or Ruby. If they've really gotten into the infrastructure as code thing they might have delved into the innards of Chef and Puppet and been exposed to those projects respective testing frameworks.

In a lot of cases, testing is seen as "something I'll get to when I become a real developer".

Read more...

Just post mortems

Earlier this week I gave a talk at Monitorama EU on psychological factors that should be considered when designing alerts.

Dave Zwieback pointed me to a great blog post of his on managing the human side of post mortems, which bookends nicely with my talk:

Imagine you had to write a postmortem containing statements like these:

We were unable to resolve the outage as quickly as we would have hoped because our decision making was impacted by extreme stress.

We spent two hours repeatedly applying the fix that worked during the previous outage, only to find out that it made no difference in this one.

We did not communicate openly about an escalating outage that was caused by our botched deployment because we thought we were about to lose our jobs.

While the above scenarios are entirely realistic, it's hard to find many postmortem write-ups that even hint at these "human factors." Their absence is, in part, due to the social stigma associated with publicly acknowledging their contribution to outages.

Dave's third example dovetails well with some of the examples in Dekker's Just Culture.

Read more...