Software Delivery in Small Batches

Hello and welcome to Small Batches with me Adam Hawkins. In each episode, I share a small batch of the theory and practices behind software delivery excellency
Topics include DevOps, lean, continuous delivery, and conversations with industry leaders. Now, let’s begin today’s episode.

There’s a phrase that’s coming up more and more in my daily work. It relates to one of my favorite questions these days: How do we know what we know?
This is a powerful question because it summons facts and knowledge about the current condition. We call that a mental model. We can test the mental model immediately in the real world.
The challenge is getting to the quantifiable and empirical facts. That’s where this phrase comes in.
That phrase is: as measured by. Here’s how to use it.

Let’s begin with an example from my day job as engineering manager for an SRE team.
We run a golden signals workshop with ourselves and other teams. The aim is answering this question: how do we know the system is working? Answering the question creates a mental model for operating and troubleshooting the system.
The workshop consists of creating tier one and tier two signals for each of the golden signals. Quick refresher: the golden signals are latency, errors, traffic, and saturation. Also, link in the show notes for a previous podcast episode on the golden signals.
The tier one signals are candidates for monitoring, use in SLIs, and the stuff to chart on the team’s primary dashboards. The tier two signals are for supplementary troubleshooting and diagnostic dashboards.
We encourage participants to throw stickie notes onto a Miro board, then work to refine them all into a coherent mental model. Here’s how the phrase “as measured by” drives that exercise.
Let’s say someone adds a sticky note for “HTTP Requests” under “Traffic”. Great start, but no where near precise enough. This phrase is wonderful because it helps newcomers navigate operations and systems architecture.
This leads to questions like: “Can you show me where HTTP requests are measured in this system diagram?” If they point to the application and not the load balancer, then that’s an opportunity to explain the gap in understanding. It’s a chance to probe the path requests actually take to become “traffic” for that service.
Vague stickie like “HTTP requests” becomes one or more stickies like “Total HTTP requests as measured by the load balancer” and “Total HTTP requests as measured by each application instance”. Now the mental model is more clear.
We continue asking these questions to refine the stickies. We know we’re done when they include “as measured by”. That typically points to how and where to collect the telemetry in the system.
That’s my personal example. I’ll close out the episode with where I learned this phrase.
It comes from John Doerr’s book “Measure what Matters” on using OKRs. The use of “as measured by” notches in wonderfully there too.
OKRs stands for Objective and Key Results. The objective is a target-condition. The Key Results are how you know you’ve met the target-condition. The idea is describe the what or where without the how.
OKRS typically read like this: I will (Objective) as measured by (Key Results).
Here’s an example written by someone who wants to run a 10K.
Objective: Run a 10K in under 50 minutes by June; as measured by these key results:
1. Go for a run 3x/week for at least 30 minutes.
2. Increase distance of run by 1 mile every week.
3. Increase mile speed by 5 seconds every week.
Using this phrase guides you towards measurable and objective outcomes, instead of open-end and subjective assessments.
Here is a rephrased example from the book for this podcast episode.
Objective: Introduce the language of empirical thinking as measured by three key results.
1. Finish this episode in under five minutes.
2. Show two examples of the phrase “as measured by”
3. Provide three links on empirical thinking in the show notes

All right that’s all for this batch. Head over to https://SmallBatches.fm/90 for links to recommended self-study on empirical thinking and ways to support the show.
I hope to have you back again for next episode. So until then, happy shipping!