My software engineering team and I were recently laid off from the digital advertising firm where I worked for the last 3+ years. It was not a surprise. Custom software was never the agency’s value proposition. After the first year and a half, when our initial projects were complete and stable, the signs were clear that we didn’t have much future support within the organization, which was focused on redefining itself even as it coped with hemorrhaging revenue. The layoff was a positive thing for me, fishing me out of the proverbial gradually-heating frog-pot. And it allowed me to distill some lessons from the decline and fall of in-house engineering at the agency.
Software engineers are expensive. If the software they build doesn’t earn or save at least as much money as their combined salaries, they’re in trouble — especially if their product is not at the core of the company’s real value proposition. The first of my team’s projects was to resurrect and improve a custom portal that served one unique segment of the business. The finished project was celebrated (exaggeratedly, in my opinion) as saving the client team one business analyst head count, and was presented to the rest of the company as a success. In truth, the new portal’s reporting required a full time Business Intelligence analyst (so the head we had “saved” just shifted to another department), and it took a team of three programmers, one QA analyst, one DevOps engineer and their management overhead (all of whom were far costlier than the business analyst role) almost a year to complete. The software boasted some interesting features and integrations into the agency’s technical ecosystem, but numerically, by any fair measure, the project was a colossal waste of money. The business group’s needs were too unique within the agency for us to use the technology to benefit other groups. And the most ironic thing was, over the course of three years, the business segment replaced one piece of unsupported custom software (the portal’s original developers were lost through an acquisition deal) with another (once my in-house engineering team was laid off). For the cost of a couple of entry-level or offshore analysts, the team could have used a combination of off-the-shelf software and manual effort to achieve much the same, for far less money.
So why did they start developing software in-house? My agency knew that technology could help them automate processes and “scale the business,” but it didn’t have a clear understanding of what it means to scale. To software engineers, scalability has a precise meaning: the ability to increase the volume of well-defined, repeatable work that a system handles within a period of time, at a lower cost per unit of work. My agency tried, as many non-technical folks do, to hand-wave around the need for “well-defined, repeatable work.” In fact, each client group had its own way of categorizing data, managing it, reporting on it, and so on. There were no standard practices or agreed-upon best-in-class solutions. If two clients were handled in any way identically, it was more coincidence than design. Technology couldn’t help the business scale, because there were no non-trivial scalable processes across client groups. Technologists can assist by holding company practices to the kind of scrutiny and rigor required by software requirements, but they are not the ones to design the business solution. It falls on the shoulders of business domain experts to define the best practices. Once the agency realized this, the effort to define these practices outlived the engineering team that could automate them.
A lot of the reasons in-house software engineering failed at the agency can be laid at the feet of fragmentation, siloing, and a culture of firefighting. The agency was aware that it needed to overhaul its identity and unify its vision. But it had to do this amid the reality of largely independent client and business groups struggling to maintain client satisfaction in the face of employee attrition, tightening budget, and aggressive, externally-imposed revenue goals. In this climate, the agency’s key players were focused on immediate-term tactics. Software development, even “agile” development, relies on business sponsorship that has a consistent, long-term, agreed-upon strategy. Software is an investment, and the chances of it paying off are proportional to the business understanding that goes into its design. My team had both annual and quarterly delivery roadmaps. We had a visible, prioritized backlog of work that we executed on two-week cycles. We gave presentations to other business groups to promote ourselves. It was telling that few outside of engineering knew of our planning artifacts, and fewer still cared to accept our invitation to contribute. Whether or not the house was truly on fire while we were planning renovations, I lacked the perspective to know. But it’s likely that the agency had adopted a firefighting habit — no, culture — and the time to breathe and take stock would never come, for the business groups and the agency as a whole. Investing in software development was extraneous to the way the agency operated.
My employment at the agency was the first time where my work was an expense, not revenue. I knew that would make it more crucial to align with the organizational vision, but I was unprepared for how much that would require me to assume the role of an educator and business strategist. My team enjoyed an unusual amount of autonomy (we defined our own processes, set up our own infrastructure using Amazon Web Services, and chose our own technology stack), and we even set our own priorities when our internal clients seemed ambivalent. But rather than being a perk, these were symptoms of an organization that wasn’t sure what to do with us over the long term. What I haven’t described (and should probably describe, in a separate post), is how grateful I am for the opportunity to build a software engineering practice from the ground up. We got several things right, either at first or through iteration and improvement. But it’s an expensive proposition. And there’s a significant common understanding required before business can successfully partner with technology.