Testing those difficult-to-reach exceptions.

19 12 2006

Summary

Generic parameters can be used in “throws” clauses. This makes it possible to define a method that can throw any exception passed to it as an argument, whilst being treated by the compiler as having only that particular exception’s type in its “throws” clause (for example, using a method declaration of the form “<T extends Throwable> methodName(T argName) throws T”).

Such a method, together with a boolean property through which test cases can turn the actual throwing of the exception on/off, can provide an easy to use, general technique for testing exception-handling paths that would otherwise be difficult or impossible to test, by adding a simple call to the method into any code that requires it.

Background

Every now and then I find myself with exception-handling code that can’t easily be tested because it is difficult or impossible to force the relevant exception to occur. Until recently I’ve taken an ad-hoc approach to testing such code, but I’ve recently adopted a more general solution, and in doing so have stumbled across a neat little feature of “generics” that I hadn’t realized before.

There are a variety of circumstances in which one can end up with code whose exception handling can’t easily be tested. If you always try to write comprehensive tests for your code, you’ve probably encountered such situations at one time or another. Sometimes it’s due to calling a method that can potentially throw a checked exception but where the cirumstances of the call are such that the exception can’t actually occur; sometimes the exception is one that the JDK defines as optional depending on the JVM implementation or underlying platform; sometimes the exception is just for truly exceptional circumstances that can’t easily be conjured up on demand. There are probably a variety of other situations where this can arise.

I’ve also encountered this with some code-coverage tools when trying to get 100% code-coverage on “synchronized” blocks, due to the byte-code having a separate path for exiting the block if any exception occurs (that is, although the source code does not have any explicit exception handling, the byte-code includes a separate path for catching any unchecked exceptions thrown during the block).

As a simple example, consider a call to one of the JDK methods that take a “charset name” argument and throw an UnsupportedEncodingException if the specified charset is not supported by the underlying JVM. If you’re specifically using one of the “charset name”s that are guaranteed to be supported on all JVMs, then the exception can never actually occur. You wouldn’t generally want to leave the UnsupportedEncodingException uncaught, as that would force all callers to cater for a checked exception that can’t occur and probably isn’t at all relevant to them (and in any case this would just shift the same problem up to the calling methods). You also wouldn’t usually want to just ignore such an exception either, because if it does occur then something is clearly wrong (and this would still leave you with an exception handling path that you can’t test, albeit one that just ignores the error). So you end up with code like the following, with a “catch” clause that you can’t easily test.

try {
    foo = bar.getBytes("US-ASCII");
} catch (UnsupportedEncodingException e) {
    // Shouldn't occur, as US-ASCII always supported.
    throw new InternalFailureRuntimeException(
        "JVM did not recognise charset name US-ASCII.", e);
}

Of course, one could just decide to not test the exception handling. For the above example, that’s probably reasonable, but for other situations it might not be. In any case, I’m always wary of code that isn’t tested (especially error-handling code, where execution of the code only happens when at least one thing has already gone wrong), and for various reasons on my current projects I always require my tests to give a clean “100%” code coverage.

Until recently I’ve taken a fairly ad-hoc approach to each such situation. Often the problem can be avoided by some combination of judicious refactoring, additional configuration or indirection facilities, and the use of suitable mocks/stubs or other such “custom” implementations for relevant method arguments. At worst, one can always introduce code to explicitly throw the required exception at some suitable point under the control of an additional boolean property, with suitable naming and documentation to indicate that this is for testing purposes only.

Inevitably, all solutions to this involve some degree of compromise to the design of the code and its public or protected interface. Tackling each situation independently results in a variety of different solutions, compromises, explanations etc, as well as needing separate analysis and decisions for each situation. So recently I’ve tried to see if I can find a simple, consistent solution for all such situations.

Solution

The most general solution would seem to be to introduce “test only” code to explicitly throw the relevant exception for those particular tests that require it. As a minimum, this involves an “if” condition controlled by some suitable boolean, as well as the actual throwing of the exception itself and suitable explanatory comments. Is it possible to move the “if” logic into a centralized method, and thus reduce this into a single self-documenting method call? Ideally one would want a single, centralized method that can optionally throw whatever type of exception is required, together with methods that test-cases can use to turn the actual throwing of the exception on and off.

To achieve this, the required exception would need to be passed to the method as an argument. How would such a method work in terms of its own “throws” clause and the calling code’s type safety? Well, let’s try to define such a method with the exception type as a generic parameter:

public static <T extends Throwable> 
        void allowTestingWithSpecifiedThrowable(final T e)
        throws T {
    if (testWithSpecifiedThrowable) {
        throw e;
    }
}

Somewhat surprisingly, this seems to be perfectly OK and it works as you would hope – the compiler treats each call to it as if its own “throws” clause specifies the particular type of Throwable that is actually being passed to it. So if you call such a method with an UnsupportedEncodingException, it is treated as a call to a method whose “throws” clause specifies that it can throw UnsupportedEncodingException, but if you call it with a ClassNotFoundException it is treated as a method whose “throws” clause specifies that it can throw ClassNotFoundException. As a result, the calling code can pass this single method any type of exception that the calling code catches or can throw, without being forced to catch or throw any other or more general exceptions.

Together with methods to turn the “testWithSpecifiedThrowable” boolean on and off, that’s the basic mechanism. The rest of the solution is straightforward, “normal” coding – such as deciding on appropriate method/property names, and whether to make the methods “static” or place them on some relevant object instance and provide methods for accessing that instance. Depending on how this is done, some care may also be needed to ensure thread-safety if this is required (in particular, to ensure that individual tests can set and use the “testWithSpecifiedThrowable” boolean without interfering with each other).

For my own implementation of this, I’ve called the relevant methods “allowTestingWithSpecifiedThrowable”, “setTestWithSpecifiedThrowable” and “resetTestWithSpecifiedThrowable”; made them static methods in a non-instantiable “TestUtilities” class; and declared them all as synchronized against the class itself.

With this in place, the exception handling of the preceding example can be made testable by adding a suitable call to the “allowTestingWithSpecifiedThrowable” method:

try {
    foo = bar.getBytes("US-ASCII");
    TestUtilities.allowTestingWithSpecifiedThrowable(
        new UnsupportedEncodingException("Testing"));
} catch (UnsupportedEncodingException e) {
    // Shouldn't occur (except in testing), 
    // as US-ASCII always supported.
    throw new InternalFailureRuntimeException(
        "JVM did not recognise charset name US-ASCII.", e);
}

The tests of the method’s normal behaviour remain unchanged, but the above code’s exception handling can now be tested using code of the following form:

synchronized (TestUtilities.class) {
    TestUtilities.setTestWithSpecifiedThrowable();
    try {
        ... call the code and check that it correctly
        handles the UnsupportedEncodingException 
        as expected...
    } finally {
        TestUtilities.resetTestWithSpecifiedThrowable();
    }
}

All other such “impossible” exceptions and hard-to-reach exception-handling paths can be tested in exactly the same way, by adding a simple call to “allowTestingWithSpecifiedThrowable” (with a suitable example exception) at the end of the method’s normal processing (immediately before the relevant “catch” clause).

In addition, to simplify the common case where I just want to test a method’s handling of any arbitrary runtime exception, I’ve also added methods into my “TestUtilities” class to optionally throw an ExplicitlyRequestedRuntimeException, with this being a simple subclass of RuntimeException. In particular, I use this to test the exception-handling exit from all synchronized blocks, to ensure that code-coverage tools don’t mistakenly report such paths as untested.

Remaining issues

The only remaining issue with this implementation is that there is no way to turn the exception throwing “on” for testing of one method (or one particular exception within it) without it also being “on” for any and all other uses of it that happen to occur within the same test. In practice this doesn’t appear to be a problem, as relatively few methods require the use of this facility, and most individual tests are small enough and specific enough to not encounter combinations of such methods. If it does prove to be a problem in future, one solution would be to introduce some kind of “identifier” into the TestUtilities methods so as to be able to turn the exception-throwing on and off for individual named situations.

Conclusion

Is this all overkill? Probably. But now that it is in place, I’ve found it to be a clean, consistent and easy way to enable testing of a variety of otherwise difficult-to-test exception-handling code.

I guess it’s not that much simpler than coding the “if…throw” directly anywhere it’s required, but it does reduce this to a single unconditional call of a specific named method. As well as being a bit simpler/shorter, this makes it relatively self-documenting (or at least lets me document it all in the TestUtilities class rather than in each place it is used). Usages are also easy to search for. In practice I’ve definitely found this to be simpler and more explicit than the various ad-hoc approaches I’d been using before.

More generally I think it’s really neat that one can use a generic parameter in a method’s “throws” clause.

Advertisements

Actions

Information

One response

17 01 2007
closingbraces

An interesting discussion about how best to handle “impossible” exceptions has developed at http://www.artima.com/forums/flat.jsp?forum=106&thread=191686 (it’s diverged a bit into a more general discussion about Java error handling, including RuntimeException vs. Error, assertions, logging etc etc – but it’s always intersting to see the diversity of opinion on such things).

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s




%d bloggers like this: