People are attracted to easy answers for difficult problems. We often refer these "easy answers" as fads once they gain popularity and some level of authority. The management fads are quite popular in our IT community for the simple reason that they attempt to answer the old problem of creating hyper productivity.
Many IT organizations are mired in problems creating results. If only those pesky developers would deliver what I want! Expectations turn into disappointments when the time lines are not what companies what, or when companies can't fit their dreams into the budgets that they desire.
Agile! Yes, that will rocket our organization to unparalleled productivity. It's also very simple. I can see it fits on a pamphlet and it tells you what to do. Besides, they have Scrum Master courses, where within days you can now turn your organization into a well-oiled machine. All the experts say so, and we can hire Agile coaches who will come and rescue us if we get off the message.
Ok, so you are "doing" the Agile. You follow all the rules, and have your Scrums and Retrospectives, and Planning, and you tally up all the points and doodads. Cool, there is a graph that now tell exactly when we are going to be done! This is what management has been asking for all the time.
When the magic wears off then it does not look all that special. It sort of kind of works, but so did other things, sorta kinda. And, overall it probably is an improvement if you were stuck in Big Upfront Design. But, you can probably get the same just by having Medium Upfront Design, some work list, high level estimates that you keep updating, and some regular check point.
I remember tracking work on a spread sheet that would calculate the delivery date based on the task estimates ten years ago. This was the early days of Agile awareness for the greater public. However, any developer worth some gravy would already know the tools to be successful - hence the spread sheet. It's just that when you suddenly throw in a layer of bureaucracy and a Project Management Office in the mix when things usually take a nose dive. Welcome loss of sanity.
Suddenly some publicized Agile stuff starts to sound very good because it is marketable. You can sell that to a manager in a nice package, and you can give your graphs and velocity numbers to them, and boast about great progress. Yay! That's the antidote against PMO and layers of confusion!
But, I must admit that I am tired of the masquerade. I completely understand the Agile principle because that's what competent developers do on a daily basis, and have done so for a long time. It is just that when all those things that developers do to make things successful is marginalized, externalized and parcel wrapped into a "management fad" that the basic ideas of building software systems get lost somewhere. It's like a regression more than advancement when you start reciting Scrum commandments as the only gospel.
The problem is that fads provide easy and attractive answers. Unfortunately, you can only deliver successful stuff with great people. It's not the amount of people, but the kind of people you have. You still have to do the hard work, put the hours in, and be good at great many things. People are inconsistent, and they do not follow rules. Even if they follow rules, they follow them differently each time, with variable results. If something gets on the way, people stop doing it.
Because people are the way they are, you need adaptability, some light weight process, and interaction with feedback. That paired with good people will give you results. That same thing with wrong people hardly ever will. In fact, the process probably does not matter much. You can succeed with waterfall consistently if you have the right people.