In an interview about his book with Cédric Beust, Next Generation Java Testing: TestNT and Advanced Concepts, Hani Suleiman says:
“I’m fairly strongly against the use of mocks for Java EE constructs. These APIs are often complicated and come with whole swathes of tests to verify compliance. End users never see what a pain in the ass it actually is to certify a product as compatible to a given EE API. Mock implementations on the other hand are most certainly not certified. The danger of using them is that over time, people start implementing more and more of the API, its more code that can go wrong, and more code that’s written just for the sake of making your tests look good. Increasingly, it becomes more and more divorced from the real implementation.”
But actually I strongly agree with it, and it’s one of the motivations behind ObMimic.
Yes, if you write your own stubs or mocks or use a general-purpose “mocking” tool, it can be extremely difficult to accurately simulate or predict the real behaviour of an API, and over time you’re likely to encounter more and more of the API. You can also find yourself needing more and more scaffolding and instrumentation to serve the needs of your tests. So it can be problematic and uneconomical to do this as part of an individual application’s testing. Even when it remains simple and doesn’t grow over time, it’s still additional thought and effort for each individual application, and very easy to get wrong. Whilst it’s all theoretically possible, and can seem very simple at the start, the long-term economics of it don’t look good.
But it looks rather different if the necessary facilities are all provided for you by a specialist library developed by someone else. Then it’s up to them to figure out all the quirks of the API and provide complete and accurate coverage, and all you have to do in individual applications is to use it.
This is much like the need for a “container” or other such API implementations in the first place. For example, given the Servlet API, it’s neither feasible nor economic for each application to implement its own servlet container, but it’s perfectly reasonable to have separate, dedicated projects that produce servlet containers that everybody else just uses.
My own opinion is that the same goes for test-doubles for APIs such as the Servlet API: it’s not worth everyone doing this half-heartedly themselves, but it is worth somebody doing it well and producing a comprehensive, high-quality library that everyone else can just use.
Of course, this only works if the resulting library is complete enough and of good enough quality for you to be able to rely on it. This points to the kind of criteria on which to judge such suites of test-doubles:
- How complete is the API coverage?
- How accurate is the API simulation?
- How configurable are the test-doubles?
- How examinable are the test-doubles?
- How well documented is it?
- How easy is it to use?
- What extra features to support testing does it provide? (e.g. strict or configurable validation of API calls, tracking the API calls made, configurable for different versions of the API etc).
- What dependencies does it have? (e.g. is it limited to only being used with specific tools or frameworks or only in certain scenarios, or is it more generally applicable).
Unfortunately, we don’t seem to have many API-specific libraries of test-doubles at the moment, and in my own limited experience those that we do have aren’t generally good enough.
That’s understandable, as it’s a huge amount of work to do this well for any substantial API that wasn’t written with testing in mind. Especially for APIs as complex, imperfect and subject to change as some of the older Java EE APIs.
Apart from my own ObMimic library for the Servlet API, I’m aware of some other attempts at doing this for the Servlet API, such as HttpUnit’s ServletUnit and the Spring framework’s org.springframework.mock.web package. However, in general these tend to be somewhat incomplete, inadequately documented, and lacking in configurability and test-instrumentation. Some are also outdated or defunct, limited to a particular web-app framework, or are a very minor and secondary component within a broader product that has a rather different purpose and priorities (and is thus unlikely to get much attention or maintenance).
In terms of other APIs, I’m aware of MockEJB for EJB, Mock Javamail for JavaMail, and a few such libraries for JNDI. There’s also a discussion of this issue in James Strachan’s blog article “Mocking out protocols and services is damn useful” (though some of the solutions mentioned are lightweight “real” implementations rather than test-doubles as such). But that seems to be about it.
Does anyone know any more? Or have any general views on the quality and suitability of any of these libraries?
As an ideal, I’d like to see the provision of a suitable library of any necessary test-doubles as a mandatory part of any Java API, in the same way that the JCP demands not just a specification but also a reference implementation and a compatibility kit.
That sounds like a big extra burden on API development. However, most new APIs ought to be designed so that test-doubles aren’t generally necessary in the first place, or can be relatively simple. For example, EJB 3.0 has far less need for this sort of thing than EJB 2.0, due to being more “POJO”-based. As another example, I believe the new JSR 310 Date and Time API is being designed so that the API itself will allow you to “stop” or control the time for testing purposes (for example, see slides 89-91 of Stephen Colebourne’s Javopolis 2007 presentation on JSR 310).
More generally, if this was always tackled as an intrinsic part of each API’s design then it ought to result in APIs that are more amenable to testing, and developing any test-doubles that are still necessary for such an API ought to be far simpler than trying to provide this retrospectively for an API that has ignored this issue. Having any necessary test-doubles should also be helpful in the development and testing of real implementations. In any case, this ought to be a more efficient division of labour than leaving everybody to hack their own way around the absence of such facilities.
As an absolute ideal, I’d want the resulting libraries of API-specific test-doubles to all take a similar form, with common features and facilities, terminology, naming conventions, usage patterns etc. But that’s probably getting into the realm of fantasy.