I often hear this question from professionals in any field who have participated in building, maintaining, or migrating software. Sometimes they ask hopefully, seeking a solution to the problems they experience, while in other cases, they ask with resignation, knowing that there is nothing that can avoid the inherent difficulties of working with technology.
Unfortunately, the numbers confirm that technology projects are often very complex, and this is a widespread issue. For instance, in the United States in 2014, 175,000 technology projects were evaluated, and the findings revealed that only 16.2% of them were completed on time and within budget. The percentage did not take into account product quality or project scope because the result would have been nearly 0%.
Humanity has not yet found the best way to develop software
If everyone struggles when building software, it stands to reason that the cause must affect everyone. This is why the best answer I've heard is: "Humanity has not yet discovered the best way to develop software; we are still experimenting and learning from our mistakes." While humanity has been building houses for centuries and still makes errors, the frequency of errors has decreased. In contrast, humanity has only been building software for about 70 years, so it's no surprise that errors are still commonplace.
So, what can you do? Learn to live with the uncertainty and complexity of software, not only in projects but also in business strategies that depend on software. Agile frameworks are a step in that direction and are the best alternative we have come up with, but they alone cannot solve all problems. There is still much to discover, so remember this when starting a technology project.