How to educate a development manager about the difficulties of software design?

I have had a few development managers who don't seem to understand or appreciate the difficulties of software design and implementation.

Such managers believe that processes and methodologies completely solve the problem and I have a tough time explaining to them that it is not so and that you cannot read a book on the latest process fad and hope to get results by applying them as is.

The latest frustration I have is to convince my manager to (a) Give me requirements not piece-meal but a larger set as far as possible. (b) Give my team lead time to think about how to design, thrash out a few alternatives, work out an implementation sketch, to plan out the tasks etc.

The frustrations are compounded because of Agile methodology and the interpretation of it that says not to do up-front design (as against BIG up-front design in Waterfall), product owner can change requirements at any time and so son.

So far I haven't had much success and have to put up with the resulting frustrations. Can you give me some arguments that can convince such managers?

EDIT-1: Retrospectives are done, though not always at the end of every sprint, and the problems are brought up. But as I mentioned, my manager doesn't appreciate the need for design lead time and the frustrations with piece-meal requirements.

EDIT-2 I don't have a problem with changing requirements. I understand that it will be so, but imagine this: You want a small feature to begin with and then you keep adding more around it. After a few iterations, the design cannot handle it anymore and a redesign (not refactoring) is required. This could have been solved better with an upfront design in the first place, had the related features been investigated together. Its not BDUF, its the natural way of doing it (what I call software engineering common sense).

My manager doesn't understand why I ask for time to redesign (a few times I just call it refactoring so that it fits the Agile way of doing it, but it really is redesign) and not developing and demoing new features.

Solution

Agile Simple Design doesn't mean don't do ANY design/architecture up front. It means do the minimal design up front, so that you will not pay a horrible price for reasonable change requests.

Scott Ambler talks about Change Cases - http://www.agilemodeling.com/artifacts/changeCase.htm James Coplien talks about Agile Architecture - http://www.infoq.com/presentations/Agile-Architecture-Is-Not-Fragile-Architecture-James-Coplien-Kevlin-Henney http://blog.jaoo.dk/2009/03/04/handling-architecture-in-the-agile-world/

The art/craft in all of this is in how to slice the architecture in a way that allows:

relatively fast convergence on overall architecture/infrastructure - on the order of days per months of estimated development time.
developing "just enough" architecture/infrastructure per each feature/requirement
doing the right balance of preparations for the future compared to focus on the features of today.

Its important that your Product Owner is aware of all of this balancing act as well, and you work collaboratively. He should understand that if you disregard all thinking for the future, each change will be very costly. There is a price to be paid for flexibility.

Its btw very similar to investment in QA and test automation. You pay something now, that will pay off only after X times you test the code. if the code never changes it was a waste of effort. but everyone knows that most code changes...