Software Delivery in Small Batches

[00:00:00] Hello and welcome. I'm your host, Adam Hawkins. In each episode, I present a small batch, with theory and practices behind building a high velocity software organization. Topics include dev ops, lean software architecture, continuous delivery, and conversations with industry leaders. Now let's begin today's episode.

[00:00:26] I'm doing something different on small batches. I've put together a series of episodes called the salt side Chronicles. These episodes tell the story of business, agility, technical debt, the fabled ground up rewrite and the proudest and most frustrating moments in my career. It's a case study on the different aspects of software delivery from team typologies, to software architecture and business agility.

[00:00:51] This is a five-part series, a new episode. We'll drop each weekday. Think of this as my Christmas present to all of the. New interviews and solo episodes, we'll be back in 2021. So this series of episodes is the last you'll hear from small batches in 2020. Now with that out of the way, let's begin our journey through the salt side Chronicles.

[00:01:17] I started working for salt side in 2014 as a general purpose for redeveloper. I believe the company had been around about two years at that. point. They're part of the development team was quite small. We all shared a single room in one office. There were about a dozen developers about half that on the product side, salt side, main classified sites for buying and selling mostly secondhand goods in developing markets.

[00:01:42] Their business strategy was to capture the market by offering a high quality and free classified service, and then roll out paid and premium services when their position allowed. Eventually they'd be the go-to classified site for that. vpuntry. Also note that they entered markets, where there was not strong existing competition.

[00:02:00] They launched their first site in Sri Lanka called Ekman. Well, before I joined their strategy was working well in Sri Lanka. They were in the top 10 sites nationwide. Also note that the top 10 sites are relatively consistent across countries. Google and Facebook are always in the top along to local news site and at classified site.

[00:02:19] So salt site broke into the top 10 and this was great news, for them. That guaranteed. The site received constant traffic and captured more market share salt side was not offering paid or premium service. At this point. This startup was still in the growth phase fault side plan to launch two new markets right around when I joined this was tauntaun in Ghana and pick Roy and Bangladesh.

[00:02:43] These were both great indicators for the company. One thing that always surprised me about this company was the complexity of the business. domain. I always thought that, well, you just have a title description, some photos and contact information. What else do you need? Well, that wasn't the case. The problem domain was much more complex than that.

[00:03:04] It took me years to truly understand the depths of the domain and why it was that way. Now, I don't know if they're part owners, executives, or engineers set out to create the system that they did, but the outcome is highly interesting to me. I wish I could have been there for the Genesis moments that led to the system I started working with, but I digress a bit for now.

[00:03:25] We'll definitely come back to this. You see the engineering team created what you could call a CMS for classified sites, all three of these markets and any future markets, all use the same code base and the same servers. The quote market functionality came through what was called quote config. This config was the heart of the systems complexity. It included all the locations for each market, the categories, and most importantly, all the details of what required an ad to be posted what fields are required, but to call those fields what units in those fields could have. And the list goes on and on this led to what I called the quote multi-product product.

[00:04:07] This is different from multitenant because each tenant would share the same logic, just maybe different infrastructure, salt side was different in the sense that every bit of internal business logic required at least two parameters, the config meaning the market and all the config just mentioned. And the entities in question.You see config was pervasive throughout the entire system.

[00:04:29] It was truly global in the sense that nothing could happen without config request processing couldn't even begin without the. config. My guess is the team set out to build something like this, to give product owners and market managers more control over the product in their market. This is a great idea from the dev team.

[00:04:46] Stay out of the critical path and allow people on the front lines to tweak the knobs. So if a team from market a wanted to customize it, well, then just point them to the Edmund dashboard for market a no need to contact the engineering team. You know, just self service. Well that grand vision never materialized. Instead engineering what's on the critical path for any of these config changes. That's much to the chagrin other product owners, because they were frequent important to the business and a pain in the ass for the engineering team. That's a recipe for a bottlenec. If I've ever heard one, put a flag in this topic because we will definitely revisit it in future episodes.

[00:05:23] So for now, just know that salt side was a classifieds business that operated at different sites on a quasi CMS, things worked in the business was progressing. Let's move forward in time. Now my first real task was to add the first paid feature to the product. This required integrating with all the existing code and in creating a new service to handle payments.

[00:05:44] This would be a new service because the bounded context was entirely different than the existing. ones. Plus it had different access requirements. There was the user-facing payment flows and the internal facing admin dashboard. Also each market had different payment providers requiring different integrations.

[00:06:03] All this made it much easier to do as a Greenfield project, instead of building it into the existing code basis, the payment service was also separated perhaps by magic from the config requirement. This was due to the fact that payments service did not need to know about ads or any other entities. It just needed to know the market.

[00:06:22] So the user facing apps could query it for payment methods. This also has changed the user facing apps a bit. They assume that config was globally available. State payment providers were different. It had to be quarried and loaded explicitly when needed. So the payment service launched and the first paid features went into.

[00:06:41] production. This was a milestone for the company because it was the first time they had ever had any revenue. I felt lucky to be the engineer behind this feature that moved the company along the way to profitability. All right. So things are still moving in the right direction for salt side. What's the next step?

[00:06:59] The next step was a mobile app. Recall the timeline. This is right around the end of 2014. You may be thinking, how did you not have an app? Well, the answer is a business. Didn't need one. The market salt side operate in we're still adopting smartphones, feature phones and dumb phones. Like those old Nokia brick phones were prevalent.

[00:07:21] Mobile internet access. Wasn't that great either. Plus smartphones were simply a prohibitively expensive in these markets, but times were changing smartphones. Thanks to Google. and Android were penetrating developing market. Internet access improved and more importantly, competitors had mobile apps. And the fact of the matter is that people just seem to like apps better.

[00:07:44] They provided a nicer experience for classifieds products. Users could take a photo on their phone and immediately post their ad. Compare that to posting an ad on a computer users would use their phone to take the photos and copy the photo to the computer and then create the ad on their browser So far for sure.

[00:08:04] I know, I feel that way whenever I had to use classified sites and I don't think other people are different, the CEO announced to the development team. The next objective was to launch a mobile app. This scared the hell out of me. I'll talk much more about this in a later episode. So let me give you the highlights for now.

[00:08:23] The essential problem was that there was no API. You see the existing user facing applications where service side, rendered. The various backends. And I use that term loosely communicated through a common library and shared database. There was no API for core business functions, like getting config or posting an ad, or even searching for ads.

[00:08:44] This was a mismatch of things that barely fit for the current purpose. Let alone for a mobile app. In short, there was just no way we were typically ready for supporting an app from the engineering side, nor from the product design side. So what to do. The business case for the app is clear and engineering can't deliver without a major architectural changes.

[00:09:05] So there we were just stuck. We certainly didn't want to cross the Rubicon of a rewrite. Instead, we decided to create a new shell around all the existing bits that could be called through an API. This required thinking about how data flows through the system, how config was handled and represented and a ton of other stuff.

[00:09:26] We would build this shell as a translation layer and then figure out how to hook it up to everything else. This was more or less my responsibility. There were no clear titles in salt side at this point or anything like that. We were only something like a dozen engineers told at the time. So I effectively stepped into some mismatch of lead developer and architect role for what the system was evolving into.

[00:09:49] As far as I could tell that role hadn't existed up until this point. Even if it was just in name only, I think that's a factor in how the system became what it did. There wasn't a person with a high level vision for the architecture and its fitness for current and future business requirements. There wasn't a person who could connect the product owners requirements to the underlying software and short.

[00:10:14] There was no one in the architect's elevator. That's a reference to a specific book. Find a link to that one in the. description. So even if we solved all the technical problems, we did not have the skills required to build a mobile app on staff. We were a small development team who had only built web applications.

[00:10:33] So it was contracted out to a development firm in Sweden. I was the contact point for their development team. I'm not sure what your experience has been with these type of projects, but this one did not go well because the requirements were not known upfront. You see building this mobile app was really about discovery for us.

[00:10:54] We needed a tight collaboration with product owners to actually create the UX and higher integration with the development team to make it all happen. We weren't able to get something going, but it certainly not going to complete in the contractor model. However, this did allow salt side to complete what was effectively a very expensive spike.

[00:11:12] We discovered some requirements from the app side and gain more insight into the API side. Then something happened out of the blue that changed salt side, forever salt side merged with another subsidiary of the parent company. This doubled the size of the engineering team and provided the much needed engineering skills in house to build a mobile apps.

[00:11:31] We went from having zero mobile engineers. So having enough engineers to build an Android and even an iOS app, we got more designers too, and more importantly, the co-founder of that company became salt. Side's first. CTO. Deleted engineers of the web and mobile applications of that company became my peers.

[00:11:50] This put me as a lead of the backend team. We handled all the APIs and infrastructure that the web and mobile teams needed to build their apps. The mobile team handled Android and iOS. The web team handled. These are facing and internal facing web applications. All three of us reported directly to the CTO.

[00:12:10] Naturally mobile app development moved in-house based off the prior art created by the local development firm. Next point is a key part of our story. Now, I don't know exactly when or how this happened, but what was intended to be an Android app transformed into a complete ground-up rewrite of the entire product.

[00:12:32] You see, initially the plan was to just launch an Android app and build the API support. that. morphed into launching an Android app, an iOS app, and completely redesigned desktop and mobile web, and even update to the internal user facing admin application, salt side, effectively halted any new features and put 100% of their resources into launching quote salt side 2.0, that would define the company for years to come more on this in part three of a series, which dives into the business and product side that resulted in the ground up.

[00:13:04] rewrite. Let me talk a little bit about the platform team's role in the rewrite and what we were dealing with salt side was a Ruby on rails development team. And unfortunately they fell into all the traps that most rails applications do just search for monolithic rails application, and you'll find plenty of horror stories.

[00:13:24] Part The salt side. The problem was the lack of internal boundaries and architectural consistency. There were a handful of services at this point too. There was a payment provider service I mentioned earlier. There was a monolith that served the user-facing desktop web application and a ton of other internal business logic.

[00:13:43] There was a mobile web service that served the feature phone and mobile web website. It communicated back to the monolith through a shared code library instead of defined network API APIs. Then there was the config surface, which was basically a database as a service. There was also the admin service that handled the all-important internal ad review process and the customer support application for internal users.

[00:14:08] This service communicated with the monolith through a shared database. Plus there was a single instance of all these services that serve traffic for all other markets that made scaling weird and operations weird because of one highly active market could crash the others and believe me, that did happen.

[00:14:26] My job as the lead of the platform team was to define a single API that could be used to build all these facing applications and the set of services to support internal operations. Moreover, we had to architect it in such a way that allowed the backend team to grow, iterate and scale the system support current and future market requirements.

[00:14:46] This meant completely reinvisioning the current set of highly coupled services into something more akin to a distributed system. Episode four in this series takes a deep dive into this effort. It's a case study in its own, right on defining service boundaries and contexts. It's hard to say exactly when the project started, but it was clear when it ended.

[00:15:08] This whole thing took about nine months. It was the biggest challenge of my career to date, and it was also the most rewarding. It was sure sweet to flip the switch on the new system and see an entirely new product come online. Everyone in the company was overjoyed on that day. The rewrite had finally finished the company.

[00:15:28] Finally had an app and more importantly, assaults item was set up to thrive for years to come by paying back mountains of technical debt. I felt my time at salt side had ended. I interviewed someone towards the end of this entire project who I did intend to replace. me. So with the rewrite complete and the next lead of a platform team lined up, it just felt natural to exit stage left.

[00:15:53] I felt I'd done all that I could for the company. Plus the rewrite had taken a severe toll on everyone involved as rewrites tend to do. Plus I was living in Sweden at the time, working for salt side and Gothenburg. I've been abroad for about four years and it felt like time to make a change in my life.

[00:16:10] I wasn't ready to go back to the states, but I was ready to go somewhere else. You know, the rewrite took about nine months. So that was all lifts, Swedish winter, as if that's such an uplifting time was spent on the rewrite. Many dark days and nights were spent hammering away in VIM and pondering software architecture.

[00:16:28] It was fitting at the projects shipped just when spring turned into summer. I was very happy to be outside in the sun with a beer summer in Ibiza called my name. So I decided to quit. I tell you though, life is funny. It can take you in directions that you never expected. It was time to tell my boss that I was resigning.

[00:16:49] I reported to the CEO at the time I checked my email. Before I walk into the office, like I always did. He had called an emergency all hands meeting at 9:00 AM. What could it be? Well, it was a bombshell. The CEO had hired a new CTO. The company was opening an office in Bangalore. India. Salt site, but now have two product development centers, the current one in Gothenberg, India and another in Bangalore.

[00:17:16] This would position the development team closer to the markets. The new CTO would be based in India. Talk about something out of left field. I think it caught everyone by total surprise, especially given that we had just finished the rewrite. I asked the CEO if I could speak to him right after the meeting, I told him I planned to turn in my resignation this morning, but this new India thing piqued my interest.

[00:17:40] You know, after all it was somewhere else, he asked if I'd meet with his CTO who was in town this week to meet everyone and discuss his fishing for the company. Well, the TLDR of it all was I agreed to move to Bangalore and work alongside the CTO to bootstrap a new product development center. The decision wasn't made that day, it did take a week or so of discussions to agree to.

[00:18:03] it. That wasn't at all. What I plant, I figured that I would resign move out of Sweden. And instead I was staying on, got sort of a promotion and moved to India. I thought I was ready for something new, but Hey, I was young and had been to India before and sounded like a fun and new challenge. That decision turned out to be a powerful one.

[00:18:25] I worked for assault site in Bangalore for another two and a half years. I met my future wife. there. I hired and trained an entire crop of engineers. I saw the architecture created during the rewrite scale to new technical and business requirements. I saw salt side succeed and grow. That was the most rewarding aspect of that time.

[00:18:46] In my career, I felt vindicated and encouraged by the long tail effects of the software architecture choices I made during the rewrite. But all that is a story for another time. So now you have the gist of the story. Let me recap this Alside Chronicles covers a story of a company who needed to launch a mobile app, but couldn't because of technical debt and eventually turned to a complete ground-up rewrite to solve that problem.

[00:19:14] It was the solution. No one wanted, but ultimately worked out. It's a case study on software delivery and business that's what's covered in subsequent episodes. There are four more episodes in this series. The next episode, episode two covers technical debt and the other factors that blocked the critical business requirements of launching an app episode three covers the scenarios and factors that ultimately culminated in the decision to complete a ground up rewrite, which just as a disclaimer, you would never ever want to do this.

[00:19:46] And I never want to do it again now, but that's out of the way we can continue. Episode four covers a logic and technical decisions behind splitting the monolith into a smaller distributed system. Like I said earlier, this is a effectively a case study in its own right episode five. The last episode is a retrospective on this entire process, through the lens of everything I've learned since then, the salt side Chronicles is part cautionary tale and success story.

[00:20:13] The company is still around and turn their first profit a few years. The person I hired to replace me is now the CTO. So things seem to be just swimming along. Nevertheless, there are valuable lessons to mind from this experience. These episodes tell the story of how I became well, me, I wouldn't be doing this podcast.

[00:20:33] If not, from my time at salt side, the people I worked with and the project sent me down this path of software delivery and business. My journey has started by experiencing it firsthand when technical debt called and Ali said, Nope, you can't do that. And blocked the business. So join me at the same time tomorrow for the next episode of the salt side chronicles, Technical debt costs that wraps up this batch is a smallbatches.fm for the show notes. Also find small batches FM on Twitter and leave your comments in the thread for this episode. More importantly, subscribe to this podcast for more episodes, just like this one. If you enjoy this episode, then tweet it or posted to your team slack rate this show on iTunes.

[00:21:18] It also supports the show and helps me produce more small batches. Well, I hope to have you back again for the next episodes. So until then happy shipping.

[00:21:31] Are you feeling stuck, trying to level up your skills to applying software then apply for my software delivery dojo. My dojo is a four week program designed to level up your skills, building, deploying and operating production systems. Each week, participants will go through a theoretical and practical exercises led by me designed to hone the skills needed for a continuous delivery. I'm offering this dojo at an amazingly affordable price. tosmall batches listeners spots are limited. So apply now at softwaredeliverydojo.com. Like the sound of small batches. This episode was produced by pods worth media. That's podsworth.com.