Software Delivery in Small Batches

[00:00:00] Hello and welcome. I'm your host Adam Hawkins. In each episode, I present a small batch, with theory and practices behind building a high velocity software organization. Topics include dev ops, lean software architecture, continuous delivery, and conversations with industry leaders. Now let's begin today's episode.

[00:00:26] Hello there. And welcome back to the salt side chronicles. So far, I've outlined the entire story in episode one and giving you a rundown of the technical challenges in episode two, this episode discusses how the project started off small and then snowballed into something much bigger.

[00:00:48] No one at salt side wanted a rewrite in the. beggining. The initial strategy was to create a new API service and then do whatever was required to integrate that with the old system. Now, imagine me waving my hands around when I say whatever, this is really a hand-waving statement, but think of this as a big adaptor between the new API and the old system, we designed a new end and then figure out how to integrate the old system.

[00:01:15] When the time came, we weren't certain that that strategy would work or how much effort it would take. This strategy did offer though a Greenfield way to design API centered from T zero. It also did not require us to solve all the problems. No one wanted to touch any of the internal customer support tools during this process, remember that we just wanted an API for the Android app.

[00:01:42] This strategy allowed us to focus solely on that. one. However, it was true that we could rewrite everything from scratch, but no one wanted that. That was the bigger challenge and larger in scope. And it was the most risky and costly option.

[00:01:59] the Android app. Wasn't just a port of the existing desktop web version.

[00:02:06] Now it was a new product or at least it was the next generation of salt side's user-facing experience. And for good reason, too. The app's business case was definitely there. If a company chooses to build an app, then they should tailor that product to the platform. No one really wants us just to port of a website to an app, this subtle change in thinking has consequences.

[00:02:31] Product managers started to think about how to move the product in different directions. There's created drift between the intended Android app and the existing web. apps. Technically this may creating our adaptor solution more challenging. I mentioned this change in thinking, because it comes up again later.

[00:02:51] Now I don't recall the exact timeline here, but there was a point after the initial contract period expired with the app development team. We stalled here because we didn't have the staff on hand to complete an Android app and the contract approach wasn't working out for. us. Just a sidebar here. It seemed that we expected that the contractors would do the work of designing the application UX for us, but that didn't really happen.

[00:03:16] We learned that we needed to be much more hands-on on the product design side. If we wanted to actually ship this application. So their project hadn't really gone into the full. swing yet. Another developer and I were kind of working on it to on and off, but business as usual was still happening, that changed when salt side hired more people, they got a new CTO, a new lead web engineer and an entire mobile team for Android and iOS, another designer and more backend engineers.

[00:03:51] Like I said, in an earlier episode, this immediately changed the game. We went from being a single development team to three that's, a web mobile and platform team. We went from trying to build an API for an Android app to building a new everything. And this is where the snowball really starts rolling. I don't think the initial plan was to rewrite everything, but it morphed into that.

[00:04:18] I don't know exactly when, but my hunch is just due to a series of choices. And here the web applications exemplify this recall that salt side had a desktop web mobile web and feature phone web version. These were all independently designed and implemented maintaining three different versions, increased load on the entire team at the trade-off of being able to specifically design each of these different implementations, but product owners and designers need to create three times the mock-ups developers had to do three times the implementation.

[00:04:53] Testers needed to three times of testing and the list goes on and there was significant frustration about this and built up over time. The siren song of the rewrite is the opportunity to solve all the problems, you know, just do it right. Well, combine that impulse with frustration and a team flush with new developers keen to build it their way.

[00:05:17] Then you have a recipe for a new Greenfield project. The team decided that the web app would be made responsive and as something that was initially deemed undesirable, but Hey times change. Okay, great. So no more maintenance of all different versions. Well, that sounds easy, right? Well, not exactly because now you need to create a new responsive design language. That falls on the product and design team to create all that stuff. So it can be used by the developers. Well, there's also a new Android and iOS app coming out too. So probably best to combine efforts on that front. Just push the snowball down the hill a bit more. All that now means that this new design language must be consistent across all platforms.

[00:06:04] Users need a consistent experience across all the platforms. It's hard to argue that though. but At what cost the cost comes in, throwing some more energy into the long project, negative feedback loop. So now the new, what app depends on the entirely new design language. All right. And you know, the new web app will also use the same API.

[00:06:25] So that's another dependency. Now the project can't complete until three major paces complete the API, the Android app, and now a fully responsive version of the web. app. You can see this siren song of the rewrites start to kick in here. And the snowball really starts rolling. These types of choices are why I'm hesitant to really call it a rewrite.

[00:06:49] It's more akin to launching an entirely new product, new apps and new design, and really a new everything, including new versions of all the internal facing tools. Truthfully, the only thing that was the same about salt site product app, we launched this thing was the data. Everything else had changed. I focused on the web team earlier, but the platform team fell victim to the siren song too.

[00:07:15] There was certainly a residual frustration around the inability to do convict changes and introduce new types of config. Their frustrations stems from the technical issues discussed in the previous episode. But anyway, the result was that product managers had a lot of config related changes in the backlog. They saw them as important and had never been able to get them through. So the rewrite was a chance to finally get them through. And that was definitely true. I had to agree with them there. Plus I also knew that if we didn't do these changes now, then there would be so much harder to do afterwards. It was a once in a lifetime opportunity to add these features.

[00:07:58] So we spent a lot of time doing config changes and introducing new types. A quick aside about types here, types for us representing something can config or really something about an ad types could be simple strings or numbers. More complex types could be string e numbs or measurements. A measurement was actually a composite type that had a number and a unit.

[00:08:20] And the measurements were convertible between units. This was actually for search purposes. There was also a more abstract composite type, which was also the most complex because it could be composed of any of the other types. Introducing new types was a challenge because types had logic around them.

[00:08:37] The backend needed to process them and the user facing apps needed to know how to display them. So you couldn't just create a new type. You actually had to build it in across the entire system. Given we were creating Greenfield user facing apps. Now was the time to get all the config changes in. Thus expanding the scope and complicating the requirements.

[00:08:57] You know, just keep pushing that snowball down the hill. You know, the train has left the station and now is all aboard the read-write train. These decisions led us to create an import service. Looking back. This is another indicator that this project is far more than just a rewrite. It's a new product. So all the decisions we made around config and the API just left us with two entirely different versions of reality.

[00:09:25] We abandoned the idea that the new system would read data from the old databases that would have been impossible. So this import service would read stuff from the old system and call it import APIs in the new system, the import service was our drunk drawer for handling all the config and data level changes.

[00:09:44] And believe me, there were certainly many inconsistencies and quirks in the old data. The old data was stored in Mongo DB with no real scheme information or versioning. So it was difficult to correctly understand older records. Well, the import service could handle that by mapping all these conditions to the proper new data structures.

[00:10:04] Mostly through a regrettably deep amount of nested ifs, but it was hacky and got the job done. All that being said, you know, it's just more snow on the snowball because these decisions for us, a team to create something new and just add more to the project, although it was satisfying to just delete this repository after we launched the system, eventually though there's no going back because everything everyone is working on is connected to or dependent.

[00:10:32] pm The rewrite project. Oh, and this project is a force, a reaction to an existential business crisis. Then there's no escape. Hatch launching is the only way out it sink or swim, just like a startup all over again. There was a window when the project could have just been an API and an app that window closed the moment.

[00:10:56] All of the new people joined. salt side. Also, we continually shot ourselves in the foot by continuously trying to just solve more and more of the problems. There's also a side of this. I haven't mentioned yet. Deep down. I knew a rewrite was the only way out of the previous system. My feeling was that we could start small by extracting the API and then update existing systems to use the.

[00:11:21] API. I had no idea. We end up on the path of literally rewriting everything from scratch and launching it all together. The project eventually crossed that Rubicon and we had to figure out how to split the monolith and create a new system for a completely different set of requirements. We being the platform team moved from a monolithic architecture to a distributed one.

[00:11:44] We moved from a single infrastructure for all markets to isolated infrastructure per market environment, and service. We also decided to adopt an RPC framework and use Docker to things largely unknown to us at the time, but just more and more on the snowball, but Hey, it's all the boards are rewrite train.

[00:12:05] So join me tomorrow for part four of the salt side chronicles, where I go deep into how the platform team architect of the new. product And mitigated the previous systemic issues see you then. that wraps up this batch Visit smallbatches.fm for the show notes. Also find small batches FM on Twitter and leave your comments in the thread for this episode.

[00:12:27] More importantly, subscribe to this podcast for more episodes, just like this one. If you enjoy this episode, then tweet it or post it to your team slack for rate this show on iTunes, it also supports the show and helps me produce more small batches. Well, I hope to Have you back again for the next episode. So until then, happy shipping,

[00:12:50] are you feeling stuck, trying to level up your skills to applying software then apply for my software delivery dojo. My dojo is a four week program designed to level up your skills, building, deploying and operating production systems. Each week, participants will go through a theoretical and practical exercises led by me designed to hone the skills needed for a continuous.

[00:13:11] I'm offering this dojo at an amazingly affordable price. to small batches listeners spots are limited. So apply now at softwaredeliverydojo.com. Like the sound of small batches. This episode was produced by pods worth media. That's podsworth.com.