Are end-to-end tests better than unit tests, in situations where both are applicable?

Question

Let me first define what I mean by unit and end-to-end tests. Let's say you have a program with a bunch of Java classes: A invokes B, which invokes C, and so on.

A unit test is a test for A mocking out B, and separately a test for B mocking out C, and so on.

An end-to-end test is a test for A that tests A and, transitively, B and C.

For simplicity, and to keep the discussion focused on the topic at hand rather than getting distracted by secondary details, let's assume that the system as a whole is stateless: you invoke the top-level (A) with an input, and you get an output. A given input has exactly one valid output.

To be clear, I am not including external systems here, like RPCs to other servers, databases, external state like the filesystem, UIs of any kind ("assert that programmatically tapping the Delete button deletes the current document"), etc. We're just talking about a bunch of classes within the same process.

Now, there are two approaches one can take:

1. Write end-to-end tests that try to cover all possible inputs and states. Write unit tests only when needed, like if a particular class is not adequately tested by the end-to-end test, or if the end-to-end test fails and you find it helpful to write a unit test to localise the bug. But in general, the goal is to have thorough end-to-end tests.

2. Write unit tests that test each class or component exhaustively. Write an end-to-end test as an afterthought, or perhaps not at all. Even if you write it, don't try to exhaustively test all possible inputs.

I prefer (1), because if the end-to-end tests pass, and are exhaustive, I know that the system as a whole works for all the cases I tested. Whereas if each class or component works correctly, there could still be bugs at the points of integration between them, which is where I read most bugs occur (sorry, I don't have a reference right now).

So, which of these has worked better for you — having thorough end-to-end tests, or having thorough unit tests? Why? Please give concrete reasons so that I, and other readers, can evaluate the answers for themselves.

If this question is a better fit for programmers.stackexchange.com, please move it there (moderators).

Answer 1

I would consider future changes and complexity to decide which one to go:

• If A's behaviour is likely fixed (says it's an API), while B & C are likely to be replaced/changed in the near future (says they are internal components that use an external library that is near its new release date), then I would go with end-to-end testing of A.
• If end-to-end testing of A will result in a very exhaustive list of test cases, or if there are too many test cases that are too hard to construct, I would go with unit testing of A, B and C.

I often find myself using a mix of both end-to-end and unit tests. End-to-end tests cover the crucial cases (which amount to about 60-70% coverage of all participating classes), while unit tests cover the rest (exceptions, very rare/deep execution paths), or I add in whenever I want to be more confident in my logic.