By the closing years of the 20th century the role of business software had moved past simply enabling back-end database tasks and into the role of enabling faster business processes. Software moved from supporting business processes to providing competitive advantage.
However this change came with some problems. The software development processes were set up to support fixed, slow changing requirements. Software was now trying to support fast-moving and constantly changing business needs.
Constantly changing business processes led to constantly changing project requirements. A string of high-profile failures dogged the industry in the late 1980s and throughout the 1990s.
In response, the phrase “you can’t develop on quicksand” was used to justify increasingly strict adherence to documented specifications – the software system was built and delivered in accordance with the specification and required changes were entered into a change log for future releases - and then Version 2 was developed to deliver the required change. I remember that "Version 2" became the answer to every change request.
Consequently, the systems delivered were never a reflection of the current business requirement.
Software developers became frustrated by this problem – characterised by the phrase “The Project was a success, but the business need died” – and they started experimenting with shorter cycle times and more fluid requirements processes.
And so the key principals of Agile were born, not as "a better way" - the previous way worked until it didn't - rather, Agile was born as a response to a change.