Software Delivery in Small Batches

[00:00:00] Adam Hawkins: Hello and welcome. I'm your host, Adam Hawkins. In each episode I present a small batch, with theory and practices behind building a high velocity software organization. Topics include dev ops, lean software architecture, continuous delivery and conversations with industry leaders. Now let's begin today's episode.

[00:00:26] Adam Hawkins: Hello everyone. welcome back to small Batches. This is the third and final part of my conversation with John Willis on Dr. Deming. John Willis is a co-author of the DevOps handbook and also the host of the profound podcast dedicated to discussing Dr. Deming's. work. We pick up the conversation with common cause and special cause variation.

[00:00:51] Adam Hawkins: This was one of the biggest takeaways from the new economics for me and it. After I've had some time to think about it, it's a really given me some food for thought on how I should work differently. So I hope it does the same for you. Enjoy part three of the conversation.

[00:01:11] Adam Hawkins: John welcome back. So we left off last time talking about like identifying what's wrong with the system, which I think is a great segway into special cause and common cause, which is something we brought up in the past. But can you explain a little bit more to listener what that idea is?

[00:01:28] John Willis: Yeah. So, it was originally created by a guy named Walter show it who was dealing with, large factories, basically you know, in the 1920s, right.

[00:01:37] John Willis: Telephones with the bus. Right. It was getting like telephones and telephone Oh my goodness. I tell them, look at it. It's awesome. Right. So he worked in the factories. I read a book on, it this town called Hawthorne, which is a fascinating story by itself. But they said that like, if you were, in like the 1920s, if you were talking about cars, you were talking about Detroit.

[00:01:55] John Willis: If you were talking about steel, you're talking about Erie. If you're talking about electronics, you would talk about Hawthorne outside of Chicago. And so this is a place where they were, they manufacture, They had like 40,000 people. By the way, if you ever heard of the Hawthorne effect, I mean, it's sort of a Mecca for a lot of different, cool stuff that was done.

[00:02:13] John Willis: And by the way, Demi did his internship here too. All right. so there's like 40,000 people at the sort of height, maybe, I don't know, 1925 basically working in some capacity to produce equipment. That would be part of a telephone factory telephones, the whole shebang, right? This is. No, it wasn't a big bell.

[00:02:32] John Willis: I think it became bell. Right? So it was hot on it. But at the time it was like 5,000. It was workers with downstream manual tests. So show it realized you're yet sort of engineering, he had statistical background. Right. A lot of cool stuff was coming out from like in the late, you know, sort of 19th century and Ludwig Boltzmann was statistical mechanics, completely applying statistics to really weird things.

[00:02:56] John Willis: Right. And so I came up with this idea, like, could we use statistical analysis to look for the causes? Of defects. Okay. Right. And the first thing he realized if you use statistics, so it's really is very sort of simple as you take all the data you create a sort of a standard deviation, three Sigma below three Sigma blue above the, sort of the mean of the data.

[00:03:20] John Willis: You know, it sort of various reason complexity, but the simplistic explanation of it. And then anything above the three, three Sigma and below the three Sigma on sort of the control card. You had to make it visible. That was his first sort of point. And it wasn't that sorry. It was to be visible in front of a screen or, but you had to put the data out of your head and into something that you can sort of mentally visualize.

[00:03:42] John Willis: So he created this control chart of this idea of like you plot the data, either within three Sigma above a blow or so the about here is like called that a assignable cause I'll just to make it simple. I'm going to teach you as the terms of Deming, renamed it to, which was special cause and common core.

[00:03:59] John Willis: So special cause was anything above or below the three Sigma and when, and when you're just trying to figure out problems and honestly, those are reasonably easiest. There They're, either black. Swans, Like there was an electrical outage and we had a ton of defects at this shutdown or start-up of the generator or whatever.

[00:04:15] John Willis: Right. So that I know what that is. Okay. We're going to try and make sure to move generator and go down next time, whatever. Right. But, or it's some sort of anomalous scenario where like, you know, somebody was clearly not trained. Right. You know, they were put in the wrong. So you can like fix those reasonably quick because they're reasonably obvious.

[00:04:35] John Willis: Right. So then the real data comes with the data in between the three segments. Right? The sort of the, And, and so here's where it gets really interesting is that's what they call sort of chemicals or data that a process that's in control, but something is outside of limits, So if something was out of control, but now it's sort of, it's in control, not even sort it seems control.

[00:04:54] John Willis: So now what you want to see, and this is where it gets counterintuitive is you want to see randomness Yeah. and, and what you don't want to see is patterns and Deming has a really good explanation. of like in one of his early sort of reports, he didn't really write books, but he wrote some like, sort of really large volumes of about statistics or, you know, sort of data correction or sort of what he calls statistical adjustment data.

[00:05:18] John Willis: That was his big thing. in early, they call it statistical adjustment data. And when he gave this analogy of like, but not valid stories, If you had this really expensive sort of coffee table and had this glass in it, it was really expensive and it broke and you need to replace it. you know, you'd want to like measure the size before you ordered it.

[00:05:35] John Willis: Right. And you would sort of, you know, measure twice cut once, right? Like, but let's take it to sort of extent that you maybe would measure it 15 times. And, then let's say the, thing is that we know from physics, there's always going to be variation and variance, right? Like there's so, and so what, if I took it 15 times and let's say one of the fifteen was some crazy number I'd look and say, whoa, like, oh, I gotta narrow Like, That was the one time that thing broke. I got another one. So you fixed that. Now you go back and the measurements just sort of bouncing. I don't know. Let's say it's, I'm totally off on my, sort of what sizing. but let's see. It's between, you know, 90 centimeters and a hundred centimeters. So this is variation, right?

[00:06:18] John Willis: You never, there's never an affinity number that is exact in measurement. and, they're all sort of bouncing all over the place. Okay. Well then I'll just take the average and what good does a tolerance? Well, they call it in manufacturing, they call it go-no-go tolerance levels. And then, but, but if I saw that like the first measurement and every measurement incrementally increased.

[00:06:39] John Willis: Right. I probably don't want to take just the average. I'd probably want to investigate that was something probably going wrong with my measurement instrument or maybe it was sort of, I was, the operator was getting sort of, you know, used to sort of be funny and say the operators getting drunk, but whatever.

[00:06:55] John Willis: Right. But that's, that's basically what statistical process control is, is, you know, looking at in-process data. And looking and trying to see where there are trends and there's, if you take any course on operations research, there's like seven to 10 patterns that you constantly look like four above to like all these different patterns, but that's, You know, and, it's served industries like, nuclear plants, instrument control, manufacturing very well. Unfortunately, we talked about this earlier. A lot of we've not really found any good grounding to use it in it. in software. And I think

[00:07:34] Adam Hawkins: why do you think that is? Because like I'm reading, I'm reading this. So just a little bit more back on the control chart, right? So the, one of the real learnings from this part of the book for me was that taking it back this to the Toyota kata and the high-velocity edge was about the improvement the systems. And now if we have the control chart, we can see the special causes in the common causes.

[00:07:53] Adam Hawkins: Okay, you got to do something about the special causes, because as you say, if they're black swans, you just have to take care of that. And then you bring everything within this balance of control, and then you have. You know, a stable process that you can apply some improvements that you can experiment on.

[00:08:06] Adam Hawkins: You can iterate, you can try to improve. And then like, thinking about that as, okay, this is how I have to approach the problem. Instead of just looking at these like arbitrary datathings. and I'm coming at this from a background of operations and thinking like, Hey, I need these control charts. Like across all different parts of like telemetry in the stack.

[00:08:24] Adam Hawkins: Like, how come we're not doing this right now?

[00:08:27] John Willis: So there's a couple of angles when I talked about like how, you know, that was his last book, you know, I think every movement needs a proper sort of the profit of movement. Dave stone for Canavin Simon Wardley for,the met, right? Like, you know what, if the minute they created it, they just went around.

[00:08:41] John Willis: Right? Like, so I think that's part of it. I think that a lot of that work has been very heavily driven in, industries that like don't sort of correlate well or overlap well with it, like healthcare instrument control, nuclear power plants. I will say this too. Right. You know, and God bless, you know, I think Steven spirit guy is a great guy.

[00:09:03] John Willis: I love him. But I've had a really long debate with him about Deming's influence in Japan and we'll just leave it at that. And then I started looking at like how lean was taught to us and be perfectly honest with you. I'm probably gonna write something about this. There's I think MIT gets a lot of the volume of what lean is and they don't really talk about Deming at all.

[00:09:26] John Willis: Not really. They don't talk about statistical prosecutorial [...]control. And I would say that if I had to create sort of a diagram of who is the biggest influences of. lean, Probably the bodies of works that came out of lean, all the sort of people that were there. and then Harvard, like had this whole weird thing about lean, you know, they sorta went all sorts of sideways.

[00:09:46] John Willis: Again, there's a longer discussion. So a large percentage of lean conversation doesn't include this as if Deming didn't exist. I dunno, if it's by accident or I, 'm hurt. I don't know what the thing is, but like you don't find any discussions and even some of them will say, yeah, I don't think they ever used that. I don't think Deming was there. Then you go to the university of Michigan and you look at our pro MoCADA and you look at those they're all about PDCA. They don't sorta jump on statistical process control, but I think the lean movement, right, honestly, I would say this right here is absolutely guilty of either consciously or subconsciously, not including Deming's impact on Toyota.

[00:10:27] John Willis: And it's probably the first time I ever said this publicly. I'm not saying it as a concern. I'm just saying, it's just sort of weird that in some cases you have to defend the people Deming's influence on and like I've, I win those battles every time I so touch on of getting the Deming prize in 1965, also, the original son of Toyota, who was the first sort of torture your own boss, teaching assistant prosecutor role with Deming in 1950. Like, and that's just some of like the evidence that he was involved and they directly were influenced, you know, and I think Shingo is a big proponent of, PDCA, you know, and driving that with his judoka and all those things.

[00:11:08] John Willis: But I just got to tell you, if I look at the bodywork at lean in, there's very few examples where people do shout outs to Deming. And in fact, the only place I find it is the MIT the university of Michigan, the shock, and you know, those guys and Robin

[00:11:25] Adam Hawkins: Just seems weird because if the goal here is to improve the processes, I mean, when I'm reading, when I read the book, you know, this was the first time that I ever really saw like a how do we measure and how to think about actually improving processes from. a Like,

[00:11:44] John Willis: This is a great text, to get my point, like you've read you, sound like you've been under a rock, like, you know, lean and you know, the agile principles and all of a sudden you sort of find out like Deming for my podcasts.

[00:11:58] John Willis: Like there's something broken. there... in a way we're teaching lean and agile in my opinion.

[00:12:04] Adam Hawkins: Well, then I'm happy that we're able to get this guy out there and broadcast the ideas and try to, you know, turn some of these rocks over and find ways that the ideas and theories can be applied to today's work. And I read the third edition of new economics. and They have a whole new chapter on. Deming now or something like that. And it's great.

[00:12:22] John Willis: I read the first, you know, the earlier version Yeah. I've been listening to books on tape, you know, so I haven't gotten fully through it or sort of updated verse, but I love his grandson, you know, talking about how, he was granted and didn't know what, how like:,"Hey grand dad how can you, why can't you be on NBC tonight? You know, he's like, I had no idea. My grandfather was famous, you know,

[00:12:44] Adam Hawkins: All right, John. Well, thank you so much for making the time to come on the show and introduce the audience to Deming and just sort of geek out about this. Guy's got a large body of work that undoubtedly I will continue to, you know, digest and share on the podcast and help to make connections with people who are interested in this.

[00:13:01] Adam Hawkins: Is there anything you'd like to pitch to the audience before we go?

[00:13:05] John Willis: You know, the thing is if, I say this a lot and I get no takers, I guess people just shy. I truly mean it. If you are sort of in some form of work and really industry, but any it role and you want as you've thought about dating or heard about Deming or wanna know more about Deming, ping me. I'd like to just have a conversation on a podcast with you. Like I don't, my goal with the podcast is not just the interview sort of quote, unquote famous people. Right. It's it's to sort of find outwhat people think about Deming in 2021.

[00:13:36] Adam Hawkins: Well, I can back that up. I reached out to you specifically after hearing that offer on the podcast.

[00:13:40] Adam Hawkins: So like, it's real, like if you want to talk about Deming, talk to John, had more than happy to, so, yeah. I look forward to, your book and keep you I'll listen to the podcast and see how it's going. I wish you luck on that. I think will be great. We look forward to your take on all this and how it relates to the work that we do.

[00:13:57] Adam Hawkins: And, just look forward to the future.

[00:14:00] John Willis: Yeah. Well, thank you for inviting me. There was a lot of fun.

[00:14:03] Adam Hawkins: My pleasure. All right, everybody. Thank you for listening.

[00:14:07] Adam Hawkins: You've just finished another episode of small batches podcast on building a high-performance software delivery organization. For more information, and to subscribe to this podcast, float to smallbatches.fm. I hope to have you back again for the next episode. So until then happy shipping,

[00:14:27] Adam Hawkins: Like the sound of Small Batches? This episode was produced by Pods Worth Media. That's podsworth.com.