Don’t you just hate tests that can’t be automated?

30 08 2008

I’ve been spoilt. Thoroughly spoilt. I’ve spent the last few years working on things that can be tested. Sure, it’s a lot of work writing good tests, but once they’re in place testing is a breeze. Run the script (or push the button), the tests run, and either it’s ok or there’s a list of problems to investigate and fix. New version? Made a change? Refactored some code? New environment? Need to prove something still works? Just run the tests again.

But now the day of reckoning has come. I’m doing some part-time work for a company that provides web-sites based on their own CMS product, but with lots of web-page design and bespoke customisation for each client. It’s very good at what it does, but testing each individual site is just a massive manual exercise. No automation at all.

It seems like a huge drain on resources, and unreliable and unrepeatable as well. Each new client or redesign leads to a fresh set of testing, and it all relies on you knowing what to test and what to look for. It’s done on a relatively undocumented and informal basis. Unsurprisingly, they seem to have lots of “regressions”, bugs they have to deal with over and over again.

What really bothers me is that I’m struggling to see any solution that would be more cost-effective.

The bulk of the testing is checking that web pages display exactly as intended, with correct layout, styles, positioning etc – compared to some mock-up pages provided by the graphic designers (which are substantial, but by no means exhaustive), and relying heavily on human judgement (e.g. “surely there’s supposed to be whitespace between those two elements when they are side-by-side… and shouldn’t that line be the same thickness as that one…”).

That sounds simple, but the pages combine variable numbers of different types of content, with lots of different options, combinations of layouts, complex criteria that decide each page’s content, many different optional elements within items, special formatting for particular elements when used in particular combinations, elements where the client can supply arbitrary HTML. The list goes on and on – the permutations are as near to infinite as makes no odds.

The end result is that even when the underlying CMS functionality is taken as being stable and already tested (which is never entirely true, of course), there’s an absolutely huge amount of visual checking of web-pages to be done.

Inspection of the CSS and HTML doesn’t even begin to give you any idea of whether it has the intended effect in all situations and no nasty surprises, let alone whether it works in different browsers and window sizes. Exact pixel-by-pixel comparison of each page’s image against an expected result might work, but there would be an absolutely enourmous number of expected results to set up, and it would all be very fragile (e.g. even the smallest change to spacing or to an element that appears everywhere would require all the “expected” images to be re-done).

So there seems little alternative other than to work through the various combinations and visually check each resulting web-page against the mock-ups (with exact comparison of individual elements where this seems appropriate). And do that for all the different combinations of elements and other features. All whilst figuring out which part of which mock-up page to look at for how each part of each page should look.

It might be ok if this was for a single web-site, or a CMS on its own, or a CMS with some simple “skins” with a limited set of variations. Then you could at least consider investing in an overall set of tests, maybe with pixel-exact checking of bitmaps against the design. It’d be a lot of work, and need a lot of maintenance, but if there was just one such set of tests for your entire business (and especially if you could adjust the “design” process to take testing into account), it might be worthwhile. At worst you could at least justify spending time on producing a good test plan for doing the manual testing.

But it isn’t like that. Instead, the styling is different for each client/web-site, and often with other bespoke features just for that particular client. It’s all basically a “one-off” exercise for each client. Having a few people informally bash away at a test site for a few days is far less work than trying to somehow automate this, or even develop and document a solid test plan – much of which would then need substantial revision for the next client.

I have a few ideas to help improve their overall approach, and I’m well aware of tools that could automate testing of the “functional” aspects of their CMS. But it’s come as a bit of a shock to find that not only do I not know of any tools to automate the “visual” aspects of the tests, but I can’t even envisage tools that could automate this (at least, not without requiring even more work than manual testing).

Does anyone else face this problem? Surely it’s a fairly common requirement to want to check the appearance of a set of web-pages against their design? Is there an obvious solution that I’m missing? Any solution at all? Or is this really something where there’s no alternative to homo sapien and the Mk1 eyeball?

In the meantime I’d better get on, I have a huge set of web pages to examine…





FindBugs finds bugs (again)

30 07 2008

FindBugs is terrific. I’ve been using it for several years now, and each new release seems to find some more mistakes in my code that were previously slipping through unnoticed.

I’d like to think I’m very careful and precise when writing code, and have the aptitude, experience and education to be reasonably good at it by now. I’m also a stickler for testing everything as comprehensively as seems feasible. So it’s rather humbling to have a tool like FindBugs pointing out silly mistakes, or reporting issues that I’d not been aware of. The first time I ran FindBugs against a large body of existing code the results were a bit of a shock!

In the early days of FindBugs, I found the genuine problems to be mixed with significant numbers of false-positives, and ended up “excluding” (i.e. turning off) lots of rules. Since then it has become progressively more precise and robust, as well as detecting more and more types of problem.

These days I run FindBugs with just a tiny number of specific “excludes”, and make sure all my code stays “clean” against that configuration. The “excludes” are mainly restricted to specific JDK or third-party interfaces and methods that I can’t do anything about.

Further new releases of FindBugs don’t usually find many new problems in the existing code, but do almost always throw up at least one thing worth looking into.

So last weekend I upgraded to FindBugs version 1.3.4, and sure enough it spotted a really silly little mistake in one particular piece of “test-case” code.

The actual problem it identified was an unnecessary “instanceof”. This turned out to be because the wrong object was being used in the “instanceof”. The code is intended to do “instanceof” checks on two different objects to see if both of them are of a particular type, but by mistake the same variable name had been used in both checks. Hence one of the objects was being examined twice (with the second examination being spotted by FindBugs as entirely superfluous), and the other not at all. If this had been in “real” code I’d have almost certainly caught it in testing, but buried away in a “helper” method within the tests themselves it has managed to survive for a couple of years without being noticed.

I guess this raises the broader issue of whether (and how) test-case code should itself be tested, but that’s one for another day (…would you then also want to test your tests of your tests…?). Anyway, thanks to FindBugs, this particular mistake has been detected and fixed before causing any harm or confusion.

Every time I find something like this it makes me think how fantastic it is to have such tools. I use PMD and CheckStyle as well, and they’ve all helped me find and fix mistakes and improve my code and my coding. I’ve learnt lots of detailed stuff from them too. But FindBugs especially has proven to be very effective whilst also being easy to use – both in Ant scripts and via its Eclipse plug-in.

If you’re writing Java code and haven’t yet tried FindBugs, it’s well worth a look.





JspC: Switching from Tomcat to Glassfish

21 07 2008

The Ant build script that I use for all of my projects includes, for web-applications, translating and compiling any JSP files. For my purposes this is just to validate the JSPs and report any syntax and compilation errors as part of the build, rather than to put pre-compiled class files into the finished web-app.

I’ve just quickly switched from using Tomcat’s JSP compiler to using Glassfish V2′s JSP compiler, and it seems worth documenting the changes involved and some of the similarities and differences.

Note that I was previously using the Tomcat 5 JSP compiler, and it didn’t seem worth upgrading this to Tomcat 6 just in order to ditch it for Glassfish, so this isn’t a like-for-like comparison – some of the fixes/changes noted might also be present in Tomcat 6.

The actual change-over was relatively painless. It’s basically the same JSP compiler – which I understand is known as “Apache Jasper 2″ – so the general nature of it and the options available are essentially the same.

In Tomcat this is provided via a “JspC” Ant task, and needs to be supplied with a classpath that includes the relevant Tomcat libraries. In contrast, Glassfish provides a “jspc” script that supplies the appropriate classpath and invokes Glassfish’s JSP compiler, passing it any supplied command-line arguments.

So switching over basically just consisted of taking out the invocation of the Tomcat-supplied “JspC” Ant task (and the corresponding set-up of its classpath), and replacing it with an Ant “exec” of the Glassfish “jspc” script with equivalent command-line arguments.

However, the Glassfish documentation for this seems a bit on the weak side. At least, I didn’t find it particularly easy to locate any definitive documentation on the command-line options for the Glassfish V2 “jspc” script. Maybe I just didn’t look in the right places. The program itself supports a “-help” option that lists its command-line options, but without much explanation. There’s a more detailed explanation of the options in the Sun Application Server 9.1 Update 2 reference manual at http://docs.sun.com/app/docs/doc/820-4046/jspc-1m, but this doesn’t entirely match the current Glassfish release (e.g. it doesn’t include the recently-added “ignoreJspFragmentErrors” option). Nevertheless, it’s the best documentation I’ve found so far. In any case, the options haven’t yet diverged much from those of Tomcat JspC, so much of the Tomcat documentation remains relevant.

I’m also a bit unsure of the exact relationship between the Tomcat and Glassfish code. They both appear to be “Apache Jasper 2″, but this doesn’t seem to exist as a product in its own right, only as a component within Tomcat. The Glassfish code is presumably a copy or fork of the Tomcat code, but with its own bug-fixes and new features, and maintained and developed as part of Glassfish. With Glassfish being the reference implementation for new JSP versions, I assume the Glassfish implementation is now the main branch going forward, even if some of the changes get incorporated into both.

To add to my uncertainty, I’m also rather confused as to whether Glassfish does or doesn’t also provide an Ant task for invoking its JSP compiler. There is an “asant” script that invokes Glassfish’s internal copy of Ant with a suitable classpath, with various targets and supporting Ant tasks. There’s also documention for previous releases of the “Sun Application Server” that show a “sun-appserv-jspc” Ant task. But the current Glassfish V2 documention doesn’t seem to list any such task amongst its “asant” targets, nor otherwise document a “jspc” or “sun-appserv-jspc” Ant task. Maybe I just didn’t find the right document. I guess I should just hunt around the Glassfish libraries for the relevant class, or try invoking it based on the previous release’s documentation. But for the moment, invoking the “jspc” script is perfectly adequate for my purposes, so I’m sticking with that unless and until I get a chance to look at this again.

A few other findings:

  • When given a complete web-application, the Tomcat 5 JspC compiler seems to process precisely those files that have a “.jsp” or “.jspx” extension. Maybe someone can enlighten me, but I can’t see anything in the Ant task’s attributes that allow it to be configured to process other file extensions. In contrast, Glassfish’s jspc script seems to automatically process all file types that are identified by the web.xml as being JSPs.
  • With the Tomcat JspC task, the JSP translation had to be followed by a separate run of “javac” to compile the resulting java source code. In contrast, the Glassfish jspc script supports a “-compile” option that carries out the compilation as part of its own processing. What’s more, I gather this uses the JSR 199 Java Compiler API for “in process” compilation if this is available (i.e. when running on JDK 6 or higher), and seems much faster as a result.
  • A slight limitation of the Glassfish jspc “-compile” option is that there doesn’t seem to be any control over where the resulting class files are written. Instead, they just get written into the same directory as the java source files. For my purposes this doesn’t matter, but if you wanted to put the class files into a specific location, or deploy them without the source code, you’d have to follow the jspc run with your own moving/copying/filtering of files as necessary.
  • I’m not particularly concerned with the exact performance of this, but subjectively the builds do seem noticeably faster since switching over to the Glassfish JspC and using its “built-in” compile instead of a separate “javac” run.
  • The Glassfish jspc script also supports a “-validate” option, which validates “.tld” and “web.xml” files against their schemas and DTDs. However, I don’t currently use this, and instead use a separate run of Glassfish’s verifier script to verify the finished web-application archive as whole.

I wonder if anyone can clarify the exact relationship between the Tomcat and Glassfish JspC implementations and the underlying “Jasper 2″? Or the exact status (and maybe classname, location, documentation etc) of any Glassfish “jspc” Ant task?





Glassfish v2 and Kaspersky 7

12 06 2008

I’ve just encountered a problem starting Glassfish V2 on an MS Windows PC, and it seems to be due to Glassfish’s JMX port being blocked by Kaspersky 7 Anti-Virus for some reason. Glassfish gives the appearance of starting successfully, but then after a while it terminates with exceptions due to a time-out whilst trying to connect to port 8686 for JMX/RMI.

There’s nothing in the Kaspersky logs to indicate that anything has been detected/blocked, but the problem goes away if Kaspersky isn’t running during the start-up. It’s also ok if Kaspersky is started after Glassfish is fully up and running – it only fails if Kaspersky is running during the start-up.

This might be something peculiar to my own set-up on this particular PC, but it sounds awfully similar to what’s being discussed in “Does Eclipse Europa support GlassFish v2 UR2″ on java.net.

So in case this helps anyone else, here’s what I’ve found, and how I’ve got around it.

The problem seems to be due to Kaspersky “traffic monitoring” of port 8686, and can be fixed by setting Kaspersky to not monitor that particular port. Unfortunately, there doesn’t seem to be any way to make Kaspersky monitor all ports except specific ones. The only way I’ve found so far to keep the monitoring in general but exclude port 8686 is to turn off “monitor all ports” and make sure that 8686 isn’t in the list of monitored ports. You’re then monitoring only those ports that are explicitly specified. This weakens the protection, but is probably acceptable if you’re behind a suitable firewall or otherwise have a known, limited set of ports that are potentially accessible.

I’ve tried playing around with other parts of the Kaspersky “web anti-virus” settings (e.g. weaken/disable “heuristic” scan, exclude localhost URLs), but I haven’t yet found anything else that fixes the problem. There might be things you can configure on Glassfish that would avoid the problem, but without any indication from Kaspersky of what it thinks it’s blocking (and why) there isn’t much to go on.

For info, this is with Glassfish V2 U1 or U2 running on Java SE 6 update 6, and Kaspersky 7.0.1.325. I didn’t previously have any problems with Glassfish V2 U1 whilst using Kaspersky 6, so I suspect this is specific to Kaspersky 7. Unless anyone knows better…





Private beta of ObMimic for out-of-container servlet testing

30 05 2008

The “ObMimic” library for out-of-container servlet testing is now being made available to a small number of users as a private “beta” release, in advance of a public beta in the next month or two.

We’re ready for a few more people to start trying it out, so if you’re interested just let me know – either via this blog’s “contact me” page or via my company e-mail address of mike-at-openbrace-dot-com.

In outline, ObMimic provides a comprehensive set of fully-configurable test doubles for the Servlet API, so that you can use normal “plain java” tools and techniques to test servlets, filters, listeners and any other code that depends on the Servlet API. We call these test doubles “mimics”, because they “mimic” the behaviour of the real object.

We see this as the ultimate set of “test doubles” for this specific API: a set of plain Java objects that completely and accurately mimic the behaviour of the “real” Servlet API objects, whilst being fully configurable and inspectable and with additional instrumentation to support both “state-based” and “interaction-based” testing.

If you find servlet code harder to test than plain Java, ObMimic might be just what you’re looking for.

With ObMimic, you can create instances of any Servlet API interface or abstract class using plain no-argument constructors; configure and inspect all relevant details of their internal state as necessary; and pass them into your code wherever Servlet API objects are needed. This makes it easy to do detailed testing of servlets, filters, listeners and other code that depends on the Servlet API, without needing a servlet container and without any of the complexities and overheads of packaging, deployment, restarts/reloads, networking etc.

ObMimic includes facilities for:

  • Setting values that are “read-only” in the Servlet API (including full programmatic control over “deployment descriptor” values and other values that are normally fixed during packaging/deployment, or that have fixed values in each servlet container).
  • Examining values that are normally “write-only” in the Servlet API (such as a response’s body content).
  • Optionally recording and retrieving details of the Servlet API calls made to each object (with ability to turn this on and off on individual objects).
  • Controlling which version of the Servlet API is simulated, with versions 2.3, 2.4 and 2.5 currently supported (for example, you can programmatically repeat a test using different Servlet API versions).
  • Detecting and reporting any calls to Servlet API methods whose handling isn’t strictly defined by the API (e.g. passing null arguments to Servlet API methods whose Javadoc doesn’t specify whether nulls are permitted or how they are handled).
  • Controlling the simulation of container-specific behaviour (i.e. where the Servlet API allows variations or leaves this open).
  • Explicitly forcing Servlet API methods to throw a checked exception (e.g. so that you can test any code that handles such exceptions).
  • Handling JNDI look-ups using a built-in, in-memory JNDI simulation.

There are no dependencies on any particular testing framework or third-party libraries (other than Java SE 5 or higher and the Servlet API itself), so you can freely use ObMimic with JUnit, TestNG or any other testing framework or tool.

In contrast to traditional “mock” or “stub” objects, ObMimic provides complete, ready-made implementations of the Servlet API interfaces and abstract classes as defined by their Javadoc. As a result, your tests don’t have to depend on your own assumptions about the Servlet API’s behaviour, and both state-based and interaction-based tests can be supported. ObMimic can even handle complex sequences of Servlet API calls, such as for session-handling, request dispatching, incorporation of “POST” body content into request parameters, notification to listeners, and other such complex interactions between Servlet API objects. It can thus be used not only for testing individual components in isolation, but also for testing more complete paths through your code and third-party libraries.

With the appropriate configuration, it’s even possible to test code that uses other frameworks on top of the Servlet API. For example, we’ve been able to use ObMimic to test “Struts 1″ code, and to run ZeroTurnaround’s JspWeaver on top of ObMimic to provide out-of-container testing of JSPs (as documented previously).

As a somewhat arbitrary example, the following code illustrates a very simple use of ObMimic to test a servlet (just to show the basics of how Servlet API objects can be created, configured and used):

import com.openbrace.obmimic.mimic.servlet.http.HttpServletRequestMimic;
import com.openbrace.obmimic.mimic.servlet.http.HttpServletResponseMimic;
import com.openbrace.obmimic.mimic.servlet.ServletConfigMimic;
import javax.servlet.Servlet;
import javax.servlet.ServletException;
import java.io.IOException;
    
...
    
/* Create a request and configure it as needed by the test. */    
HttpServletRequestMimic request = new HttpServletRequestMimic();
request.getMimicState().getRequestParameters().set("name", "foo");
request.getMimicState().getAttributes().set("bar", 123);
... further request set-up as desired ...
    
/* Create a response. */
HttpServletResponseMimic response = new HttpServletResponseMimic();
    
/*
 * Create and initialize the servlet to be tested (assumed to be a
 * class called "MyHttpServlet"), using a dummy/minimal 
 * ServletConfig.
 */
Servlet myServlet = new MyHttpServlet();
try {
    myServlet.init(new ServletConfigMimic());
} catch (ServletException e) {
    ... report that test failed with unexpected ServletException ...
}
    
/* Invoke the servlet to process the request and response. */
try {
    myServlet.service(request, response);
} catch (ServletException e) {
    ... report that test failed with unexpected ServletException ...
} catch (IOException e) {
    ... report that test failed with unexpected IOException ...
}
    
/*
 * Retrieve the response's resulting status code and body content,
 * as examples of how the resulting state of the relevant mimic 
 * instances can be examined.
 */
int statusCode 
    = response.getMimicState().getEffectiveHttpStatusCode();
String bodyContent
    = response.getMimicState().getBodyContentAsString();
... then check them as appropriate for the test ...
    

For further examples and details, refer to the previous posts “First experiments with out-of-container testing of Servlet code using ObMimic” part 1 and part 2, “Out-of-container JSP testing with ObMimic and JspWeaver”, and the related post “Mocking an API should be somebody else’s problem”.

There are also more extensive examples in ObMimic’s documentation.

ObMimic isn’t open-source, but it will have a zero-cost version (full API coverage but a few overall features disabled, such as the ability to configure the Servlet API version, control over how incorrect/ambiguous API calls are handled, and recording of API calls). There will also be a low-cost per-user “Professional” version with full functionality, and an “Enterprise” version that includes all of ObMimic’s source-code and internal tests (with an Ant build script) as well as a licence for up to 200 users.

At the moment there’s no web-site, discussion forums or bug-reporting mechanisms (all still being prepared), but ObMimic already comes with full documentation including both short and detailed “getting started” guides, “how to”s with example code, and extensive Javadoc – and for this private beta I’m providing direct support by e-mail.

Anyway, if you’d like to try out ObMimic, or have any questions or comments, or would like to be informed when there’s a more public release, just let me know via the “contact me” page or by e-mail.





Does anyone still need fax?

22 04 2008

I’m in the process of setting up phone numbers and contact details for a new IT company, and I’d started to look at fax-to-email services as we don’t want to be mucking around with phone lines or needing any particular equipment.

But then I got to thinking, why are we even bothering?

It feels almost obligatory to include a fax number in the company’s contact details, but there doesn’t seem to be any real need for this – we’re just doing it for the sake of it, and because everybody else does. The fact that we’d happily use a “pretend” fax number that actually converts everything to e-mail just seems to prove this.

Maybe I’m lacking in imagination, but if we aren’t asking anyone to send things to us by fax, it’s kind of hard to envisage situations where somebody would need or want to send faxes to us. Other than spam, of course.

Even if there might be rare occasions where for some reason we need to be sent the image of a piece of paper, how likely is it that fax would be someone’s preferred or only option (rather than, for example, scanning and e-mailing the document, or sending a copy by post)?

Maybe there are some countries with specific issues over what forms of communication are legally binding, or where bandwidth or technology constraints somehow conspire to make fax useful. Maybe there are some individuals who have access to fax facilities and like fax, but don’t have or don’t like email.

As it happens, at one point I did know someone who had a cheap fax machine as their home’s only working piece of telephony equipment, so you couldn’t phone them, you could only fax them, and if you were lucky they’d eventually notice and fax you back – but that was a decade ago, and these days even they have a proper phone that you can phone, and an email address too.

None of the above scenarios seem likely to impact this particular company. In any case, at worst we can always deal with exceptional cases individually as they arise. I can’t see anything that comes close to us needing to publish a fax number for general use.

So I’m wondering if this is just our own particular situation, or if I’ve missed something, or if fax really is now a relic of history that nobody need worry about any more.

Is there anybody out there that still likes fax? Any industries where it’s still how things are done? Any IT or otherwise “modern” companies that insist on faxes for signed contracts and the like? And most of all, is it acceptable for a high-tech IT company to not publish a fax number – or does anyone see a reason why it’s still necessary?

I’d like to think that the internet, email, image processing, legislation and business attitudes have all advanced to the point of rendering fax completely obsolete, but I dare say somebody out there will know otherwise… in the meantime our company isn’t going to be getting a fax number unless and until we find a specific need for one.





Mocking an API should be somebody else’s problem

11 03 2008

In an interview about his book with Cédric Beust, Next Generation Java Testing: TestNT and Advanced Concepts, Hani Suleiman says:

“I’m fairly strongly against the use of mocks for Java EE constructs. These APIs are often complicated and come with whole swathes of tests to verify compliance. End users never see what a pain in the ass it actually is to certify a product as compatible to a given EE API. Mock implementations on the other hand are most certainly not certified. The danger of using them is that over time, people start implementing more and more of the API, its more code that can go wrong, and more code that’s written just for the sake of making your tests look good. Increasingly, it becomes more and more divorced from the real implementation.”

You might think that I would disagree with that, in view of my current work on an ObMimic library of test-doubles for the Servlet API (in the broad xunitpatterns.com meaning of “test-double”).

But actually I strongly agree with it, and it’s one of the motivations behind ObMimic.

Yes, if you write your own stubs or mocks or use a general-purpose “mocking” tool, it can be extremely difficult to accurately simulate or predict the real behaviour of an API, and over time you’re likely to encounter more and more of the API. You can also find yourself needing more and more scaffolding and instrumentation to serve the needs of your tests. So it can be problematic and uneconomical to do this as part of an individual application’s testing. Even when it remains simple and doesn’t grow over time, it’s still additional thought and effort for each individual application, and very easy to get wrong. Whilst it’s all theoretically possible, and can seem very simple at the start, the long-term economics of it don’t look good.

But it looks rather different if the necessary facilities are all provided for you by a specialist library developed by someone else. Then it’s up to them to figure out all the quirks of the API and provide complete and accurate coverage, and all you have to do in individual applications is to use it.

This is much like the need for a “container” or other such API implementations in the first place. For example, given the Servlet API, it’s neither feasible nor economic for each application to implement its own servlet container, but it’s perfectly reasonable to have separate, dedicated projects that produce servlet containers that everybody else just uses.

My own opinion is that the same goes for test-doubles for APIs such as the Servlet API: it’s not worth everyone doing this half-heartedly themselves, but it is worth somebody doing it well and producing a comprehensive, high-quality library that everyone else can just use.

Of course, this only works if the resulting library is complete enough and of good enough quality for you to be able to rely on it. This points to the kind of criteria on which to judge such suites of test-doubles:

  • How complete is the API coverage?
  • How accurate is the API simulation?
  • How configurable are the test-doubles?
  • How examinable are the test-doubles?
  • How well documented is it?
  • How easy is it to use?
  • What extra features to support testing does it provide? (e.g. strict or configurable validation of API calls, tracking the API calls made, configurable for different versions of the API etc).
  • What dependencies does it have? (e.g. is it limited to only being used with specific tools or frameworks or only in certain scenarios, or is it more generally applicable).

Unfortunately, we don’t seem to have many API-specific libraries of test-doubles at the moment, and in my own limited experience those that we do have aren’t generally good enough.

That’s understandable, as it’s a huge amount of work to do this well for any substantial API that wasn’t written with testing in mind. Especially for APIs as complex, imperfect and subject to change as some of the older Java EE APIs.

Apart from my own ObMimic library for the Servlet API, I’m aware of some other attempts at doing this for the Servlet API, such as HttpUnit’s ServletUnit and the Spring framework’s org.springframework.mock.web package. However, in general these tend to be somewhat incomplete, inadequately documented, and lacking in configurability and test-instrumentation. Some are also outdated or defunct, limited to a particular web-app framework, or are a very minor and secondary component within a broader product that has a rather different purpose and priorities (and is thus unlikely to get much attention or maintenance).

In terms of other APIs, I’m aware of MockEJB for EJB, Mock Javamail for JavaMail, and a few such libraries for JNDI. There’s also a discussion of this issue in James Strachan’s blog article “Mocking out protocols and services is damn useful” (though some of the solutions mentioned are lightweight “real” implementations rather than test-doubles as such). But that seems to be about it.

Does anyone know any more? Or have any general views on the quality and suitability of any of these libraries?

As an ideal, I’d like to see the provision of a suitable library of any necessary test-doubles as a mandatory part of any Java API, in the same way that the JCP demands not just a specification but also a reference implementation and a compatibility kit.

That sounds like a big extra burden on API development. However, most new APIs ought to be designed so that test-doubles aren’t generally necessary in the first place, or can be relatively simple. For example, EJB 3.0 has far less need for this sort of thing than EJB 2.0, due to being more “POJO”-based. As another example, I believe the new JSR 310 Date and Time API is being designed so that the API itself will allow you to “stop” or control the time for testing purposes (for example, see slides 89-91 of Stephen Colebourne’s Javopolis 2007 presentation on JSR 310).

More generally, if this was always tackled as an intrinsic part of each API’s design then it ought to result in APIs that are more amenable to testing, and developing any test-doubles that are still necessary for such an API ought to be far simpler than trying to provide this retrospectively for an API that has ignored this issue. Having any necessary test-doubles should also be helpful in the development and testing of real implementations. In any case, this ought to be a more efficient division of labour than leaving everybody to hack their own way around the absence of such facilities.

As an absolute ideal, I’d want the resulting libraries of API-specific test-doubles to all take a similar form, with common features and facilities, terminology, naming conventions, usage patterns etc. But that’s probably getting into the realm of fantasy.








Follow

Get every new post delivered to your Inbox.