It is often said that the computational cost of the adjoint method does not increase with increasing numbers of design variables. However, it seems like there should be some computational impact from increasing the size of the problem. For example, by increasing the number of elements input in the example of ''Beam optimization Using the Adjoint Method'' the size of the FEM matricies grows and there should be more associated compute cost.
So why is it said that the cost of the adjoint is rougly invariant with increasing numbers of design variables? This seems incongruous.
In order to understand the computational cost of the adjoint method, you need to first understand how linear solves are used to compute total derivatives. You should read the OpenMDAO theory manual for a short (but still more complete) summary. You can also read the seminal paper on the various methods of computing analytic derivatives. Very briefly, when you want to compute total derivatives, you perform a linear solve (Ax=b) where the A matrix is composed of the partial derivatives.
Partial derivatives are cheap and relatively easy to compute. Its the totals that are expensive and hard to compute. If you use the direct method, then the particular linear system you form (Ax=b) needs to be solved once pre design variable (similar to finite differences). With the adjoint method, you solve a different system (A^T x = c), which only needs to be solved once per objective/constraint.
Hence, for the adjoint method, the number of linear solves you perform does not vary as you change the number of design variables. We often say that its cost is independent of the number of design variables for that reason. This is potentially a little imprecise in certain cases, when the size of the linear system is varying significantly as you change the size of the design space (i.e. if you were to make the FEM larger and have more independent elements). In that case, the cost of any one linear solve is growing but the number of linear solves is still staying the same.
Broadly speaking the number of linear solves is the most important factor in overall compute cost, so for a first approximation its is reasonable to say that the cost of an adjoint based method is independent of the size of the design space.