Slow Software Releases: Why They Happen Without CI/CD

Slow software releases are what a firm lives with when people build, test and ship code by hand. Five problems cause them: builds on one laptop, testing left to the end, release steps held in a checklist, big infrequent releases and handoffs that make code wait.

CI/CD means building and testing every change automatically, and our entry on what continuous integration and delivery do covers it in full. Here we cover what slow releases cost, when they hurt and how to tell. We do not cover how to fix them or what to buy. The build order is set out in our guide to building the pipeline in eight steps. For the aim itself, see automating builds and releases. For team models, see our piece on how platform teams and DevOps differ.

What Makes Slow Software Releases Cost So Much?

A firm pays for these delays in waiting time, risk and people, and each of five problems adds its own share. In our reading, the time goes into the path around the code more than into the coding itself. The table gives each problem, its cost and the sign that shows it.

Each problem gets its own section next.

What Goes Wrong When a Build Depends on One Laptop?

When a build depends on one laptop, only certain people can release, and nobody can reproduce a past release. Engineers build packages on their own machines. One engineer knows the commands, and no record shows how the last release was made.

The cost shows when that engineer is away. For example, a release due on Thursday may wait until Monday, or someone guesses at the steps and hopes the result matches. A firm in Bengaluru or Dubai with a small team can feel this sharply, because one absence removes a large share of the people who can ship.

Past releases suffer too. If a customer reports a fault in last quarter’s version, the team may be unable to rebuild that version with confidence. This is our reading, but the pattern is easy to spot.

What Does Testing Too Late Cost?

When testing comes last, users find the bugs, and the firm pays in time and trust to put them right. Manual tests squeeze into the final days before a release. No automated tests run on each change, so a fault written in week one surfaces in week four.

In a rushed final week, testers work through scripts under pressure. Then they may skip the cases that seem least likely to matter, and some of those cases can turn out to matter.

A bug found by a customer can cost more than a bug found by a colleague, and more again when the fix has to wait for the next release window. In our reading, the larger loss may be trust: a customer who meets a fault twice can start to doubt the next release.

Why Do Manual Release Steps Fail Late at Night?

Long manual procedures invite missed steps, and tired people can miss them. Some firms release outside working hours, in a deployment window chosen to keep customers clear of the downtime. Engineers then work through a checklist of dozens of steps at midnight.

A forgotten step is one way the failure happens. Someone skips a configuration line, runs a step twice or runs two out of order.

For example, a release team in Pune may start at ten at night so that Indian users are asleep. The team then works through the list while watching a dashboard. By the time an error appears, the team may be unable to say which of the forty steps caused it.

Why Do Big, Infrequent Releases Carry More Risk?

When many changes ship together, there is more to untangle when one of them breaks. A release that holds weeks of work makes every fault harder to trace. The team asks which of a hundred changes caused it, and the answer takes time.

Rollbacks make the problem worse. To undo one bad change, the team may have to undo unrelated fixes shipped in the same bundle. Small fixes then wait for the next window, even though they were ready weeks ago.

The cost lands on the business, which waits for improvements. A wording correction, a form fix or a new report sits finished but unreleased while the next big release fills up. Meanwhile, the pressure to include everything makes each window larger and riskier than the last.

How Do Handoffs Add Days to a Release?

Code waits for testers, then for operations, then for an approval meeting, and each wait can add days. Changes sit in a queue between teams. Approvals batch releases together. Developers may not see where their change stands.

At Signisys, we observe that much of the delay in a manual release is waiting, not working. That is a Signisys observation from the use-case page, not a statistic. The finish time may hardly move when one team works faster, since the work still waits in the queue between teams.

In our view, the gap between development and operations does the damage. Moving to the cloud or to Agile may not close it alone.

When Do Slow Software Releases Hurt the Business?

They hurt at three moments: an urgent fix, a failed release night and the loss of experienced people. Until one of these moments arrives, a slow release can pass as routine. Then the delay becomes something the business can see.

The last row is a Signisys observation from the use-case page, not a statistic. We have no figure for how many engineers leave for this reason.

How Can You Tell Whether Slow Software Releases Are Your Problem?

A yes to any question below means that problem is yours. Answer each one for your own team.

• Can only one or two engineers produce a release package?
• Are customers the ones who report bugs that testing should have caught?
• When you release, does an engineer follow a long manual list?
• Would weeks of changes wait for one release window?
• Do finished changes wait days for another team or an approval?

You can also note down two measures that DORA defines in its software delivery performance metrics: change lead time, the time from a commit to deployment in production, and failed deployment recovery time, the time to recover from a deployment that fails and requires immediate intervention. Write down both for your last three releases.

One yes means that problem deserves a closer look. Several yes answers, or no answer at all, are reasons to ask for outside help. Fixes, tools and build steps lie outside this article. When you reach the tool question, read questions to ask before picking a CI/CD tool to build a shortlist. To test your own release path, run through twelve scored checks on how releases are built and shipped.

Signisys provides DevOps and release automation services through its DevOps and CI/CD pipeline automation solution and its DevOps automation services.

Slow Software Releases FAQ: Questions Answered