In-container testing is often opposed to testing with mock objects. However, as mock objects simply mimic the behavior of the real objects, isn't the in-container testing the only way to really test the system in its' real environment?
As an partial alternative to in-container testing and mock objects, Spring provides the TestContext framework that initializes Spring nicely without needing to start up the actual application container (web application server, in my case). However, this is somewhat limited approach as it only initializes Spring-specific features while the application server -specific features are not supported. So you cannot test everything. Also, as it is not 100% the same as the default WebApplicationContext that is used in real web execution, isn't this approach a bit hackyish? Is it bad?
For in-container testing, there are at least Cactus (outdated), Jeeunit (a very little project) and JBoss Arquillian (still alpha, but looks promising). I don't see any of these projects too widely used, so is there something bad with in-container testing? The main drawback often mentioned with in-container testing is slow execution speed. However, when run in a continuous integration environment and in a relatively small project, this shouldn't be a problem.
To summarize: should we do either in-container or out-container testing and why? Would you feel bad using mock objects or an alternative initialization mechanism (as in Spring TestContext) for your integration tests?
A subnote: I recently asked about categorization of integration test, which might be relevant.
However, as mock objects simply mimic the behavior of the real objects, isn't the in-container testing the only way to really test the system in its' real environment?
I think the short answer is yes, but... I think your "categorization of integration test" question is very relevant. Unit and integration tests are both important although the serve different functions.
Unit tests are closely associated with the code, should startup and run very fast, should be run often by developers iterating on the code, and usually use mocks to a high degree. The idea is to test the code in question, not its dependencies or integration points. The problem with making the unit tests all be in-container is that they will be run less often or they will waste too much developer time.
We have isolated our in-container/integration tests elsewhere in another project with dependencies on the code projects. They are designed to closely mimic production configurations -- as much as possible. These tests take longer to setup, run longer, and are more useful to run by something like cruisecontrol. They are run by hand especially when we are closing in on a release or after development has stabilized. Often I fire up the integration tests when I'm going to lunch or a meeting. When an integration test discovers a bug, we try to write a unit test that also demonstrates the bug with the mocks -- this is often hard and can be impossible.
We usually have a couple small in-container tests with the unit tests just to make sure that the spring wiring works or to test some basic functionality but the rest of the integration testing is done in another project.
All this said, there is not a firm difference between the two. Sometimes we move unit tests that just deal with a lot of data and take a long time out to integration tests and sometimes integration tests run fast enough and are valuable enough to be included along with the code.