Some specific issues from ObMimic portability testing

7 10 2007

As mentioned in my previous Serial Monogamy post, I’ve recently been doing some portability testing on the Java code of my ObMimic library for out-of-container testing of Servlet code.

So, as promised, here are details of the specific issues I encountered, primarily as a record for my own future reference but also in case it’s any help to anyone else.

Background

First, some background:

  • The code consists of about 400 classes plus test-cases.
  • The resulting library is intended to be usable on any JRE for Java SE 5 or higher.
  • There’s nothing intrinsically platform-dependent or likely to raise any major portability issues. In particular, there’s no Swing or other such GUI code. But otherwise the code and its test-cases do use a fairly broad range of facilities. For example, there is some file handling, some charset-sensitive string processing, some URL encoding and decoding, and calls to JDK methods whose error-handling is defined as implementation-dependent.
  • Reasonable efforts have been made to keep the code fully portable (e.g. use of system properties for path and file separators, explicit use of Locales and charsets where appropriate etc).
  • All development and testing has been done on Sun JDKs on MS Windows, with testing of portability deferred until now. This approach was chosen based on the confidence gained from previous experiences with Java portability, and was judged to be the most efficient way to tackle it for this particular project.
  • The test-cases are intended to be reasonably comprehensive, and include tests of all configurable options, all error-handling code, all handling of checked exceptions etc. The EMMA code-coverage tool reports them as giving 100% code coverage.
  • My own build script for this code doesn’t need to be portable. However, one of the deliverables is an “Enterprise Edition” that includes all the source code and test-cases and its own Ant build script. This does need to be as portable as possible.
  • One potential restriction on the “Enterprise Edition” build script is that a custom Javadoc taglet is used to document one particular aspect of ObMimic’s API. In theory, such Javadoc taglets depend on “com.sun” classes in the Sun JDK’s tools.jar archive, and appear to be specific to Sun’s Javadoc tool rather than being a standard part of the Java SE platform. Hence this aspect of the build script theoretically restricts it to Sun JDKs, though the resulting library remains fully portable (and at a pinch you could always remove the taglet processing from the build if you really wanted to run the build on some other JDK). In practice, other JDKs generally claim that their Javadoc tool is fully compatible with Sun’s, and might even use the very same code. So prior to testing this, it was somewhat unclear how portable the custom “taglet” might be.

The Tests

For the moment I’m only testing on Sun, IBM and BEA JRockit JDKs for Java SE 5 and 6 plus the latest Sun JDK 7 build, on MS Windows and on a representative Linux system (actually, Ubuntu 7.04), and only on IA-32 hardware.

I’m assuming this should shake out most of the portability issues. It’s all I have readily to hand at the moment, and probably represents the majority of potential users.

I hope to extend this to Solaris and Macintosh and maybe other JDKs in future, and to other hardware as and when I can afford it. But I don’t expect many further issues once the code is fully working on both MS Windows and a Unix-based system and on JDKs from three different vendors – though I’d be interested if anyone has any experiences that suggest otherwise.

Another aim of the tests is to check that the deliverables don’t have any unexpected dependencies on my own development environment.

So the testing consists of:

  • Installing the various JDKs onto each of the relevant systems, but without all of the other tools and configuration that make up my normal development environment.
  • Putting the deliverables for each of ObMimic’s various “editions” onto each system.
  • On each system, running an Ant script that unzips/installs/configures each of the ObMimic editions, then runs the build script of the ObMimic “Enterprise Edition” (which itself builds the ObMimic library and test-cases from source, builds its Javadoc, and runs the full suite of test-cases). Then runs the test-cases against the pre-built libraries of the other ObMimic editions. And repeats this for each of the system’s JDKs in turn.

Actually, the “Enterprise Edition” build script is run using the full JDK, as it needs javac, javadoc, and the JDK’s tools.jar or equivalent, but all other tests are run using the relevant JRE on its own (to check that only a JRE is required).

The Findings

As expected, there were a few minor issues but most of the code was fine and worked first time under all of the JDKs on both MS Windows and Linux.

Even though I know it ought to work like this, it still makes me jump about like an excited little kid everytime I see it! I guess that’s what comes of past lives struggling with the joys of C/C++ macros, EBCDIC, MS Windows “thunking” and the like. Java makes it far too straightforward!

Anyway, here are the details of the few issues that I did encounter.

1. Source-code file encoding.

A few test-cases involving URL encoding/decoding and other handling of non-ASCII characters failed when the code had been compiled on Linux.

This turned out to be due to javac misreading the non-ASCII characters in the test-case source code. The actual problem is that the source files are all written using ISO-8859-1 encoding, but by default javac reads them using a default encoding that depends on the underlying platform. On MS Windows everything was being read correctly, but on Linux javac was trying to read these files as UTF-8 and was therefore misinterpreting the non-ASCII characters.

The solution was to explicitly specify the source file encoding to javac. This is done via javac’s “-encoding” option (or the corresponding “encoding” attribute of Ant’s “javac” task).

For additional safety, I also decided to limit all my Java source code files to pure 7-bit ASCII, with unicode escape codes for any non-ASCII characters (e.g. \u00A3 for the “pound sterling” character). This is perfectly adequate for my purposes, and should be the safest possible set of characters for text files. Searching for all non-ASCII characters in the code revealed only a handful of such characters, all of them in test data within test-cases.

The Sun JDK’s native2ascii tool might also be of relevance for anyone writing code in a non “Latin 1” language, but for me sticking to pure ASCII is fine.

2. Testing of methods that return unordered sequences.

Testing on the IBM JDK revealed a handful of test-cases that were checking for a specific sequence of data in a returned array, collection, iterator, enumeration or the like even where returned data is is explicitly specified as having an undefined order.

The Sun and IBM JDKs seem to fairly reliably produce different orderings for many of these cases. The Sun JDKs generally seem to give results in the order one might naively expect if forgetting that the results are unordered, but the IBM JDK generally seems to gives results in a very different order. Some of these mistakes thus slipped through the normal testing on Sun JDKs, but were picked up when tested on the IBM JDKs.

In some cases the solution was to rework the test to use a more suitable “expected result” object or comparison technique (especially as I already have a method for order-insensitive comparison of arrays). In other cases it proved simpler to just explicitly cater for each possible ordering.

It’s hard to know if all such incorrect tests have now been found. There could be more that just happen to work at the moment on the particular JDKs used. On the other hand, it’s only the tests that are wrong, not the code being tested, and the only impact is that the test is overly restrictive. So the only risk is that the test might unnecessarily fail in future or on other JDKs. For the moment that’s a risk I can live with, and I’ll fix any remaining problems as and when they actually appear.

Potentially this could also be avoided by always using an underlying collection that provides a predictable iteration order, even where this isn’t strictly required (for example, LinkedHashMap). However, that feels wrong if the relevant method’s specification explicitly defines the order as undefined, and could be misleading if callers start to take the reliable ordering for granted. This is especially true for ObMimic, where I’m simulating Servlet API methods to help test the calling code. I don’t want to provide the calling code with a predictable ordering when a normal servlet container might not. If anything, it might be better to deliberately randomise the order for each call, or at least make that a configurable option. So I’ve noted that as a possible enhancement for future versions of ObMimic.

3. All of the IBM JDK’s charsets support “encoding”.

One of the test-cases needs to use a Charset that doesn’t support encoding – that is, one whose Charset#canEncode() returns false. This failed on the IBM JDKs, due to being unable to find a suitable Charset to use.

The test-case tries to find a suitable Charset by searching through whichever Charsets are present and picking the first one it finds that doesn’t support encoding, and fails if it can’t find any such Charset. That’s fine on Sun’s JDK, where a few such charsets exist. But on the IBM JDK, every charset that is present returns true from its canEncode method, and the test therefore fails and reports that it can’t find a suitable charset to use.

Solution was to introduce a custom CharsetProvider into the test classes and have this provide a custom charset whose “canEncode” method returns false. This ensures that the test can always find such a charset, even if there none provided by the underlying JVM.

I guess I could just use this custom non-encodeable charset every time, but for some reason I feel more comfortable keeping the existing code to look through all available charsets and pick the first suitable one that it finds.

4. Javadoc “taglet” portability.

All of the JDKs handled the Javadoc “taglet” correctly.

In particular, the IBM and BEA JRockit JDKs do contain the “com.sun” classes needed by the custom “taglet”, and they had no problem compiling, testing and using the taglet.

Conclusions

Mostly, everything “just worked” as one would expect it to. The issues encountered were all pretty minor, only affected test-case code, and were easily identified and fixed.

It was worthwhile testing on both MS Windows and Linux as this revealed the source-code encoding problem, and it was worthwhile testing on both Sun and IBM JDKs as their internal implementations proved different enough to shake out a few mistakes and unjustified assumptions in the test-cases.

Some specific lessons I take from this:

  • Always specify the source-code encoding to the javac compiler (but also try to limit the source code to pure ASCII where possible, with unicode escapes for anything more exotic).
  • Whatever the other pros and cons of having comprehensive test-cases with 100% coverage, they’re mightily useful once you have them. With a comprehensive suite of tests, you can easily test things like portability (or, for example, what permissions are needed when running under a security manager). You just run the whole suite of existing tests, confident in the knowledge that this is exercising everything the code might do.
  • Whilst you’d probably assume that the Javadoc tool is a “proper” standard and part of the Java SE platform, technically it’s a Sun-specific tool within Sun’s JDK, and any custom doclets and taglets are dependent on “com.sun” classes. It seems crazy that after all this time the mechanisms for providing customised Javadoc still aren’t a standard part of the Java SE platform, but there you go. Despite this, in practice you can fairly safely regard the Javadoc tool and the “com.sun” classes as a de-facto standard. In particular the Javadoc tools in the IBM and BEA JRockit JDKs seem to be entirely compatible with Sun’s Javadoc tool. and do provide the necessary “com.sun” classes.

I’m also going to think about whether methods that return “unordered” arrays, iterators, enumerations etc ought to deliberately randomize the order of their returned elements. This would help pick out any tests that incorrectly assume a specific order. The downside is that any resulting test failures wouldn’t be entirely repeatible, which always makes things much harder. It’s also questionable whether this is worth the extra complexity and potential for errors that it would introduce into the “real” code. And it’s not something you’d want to do in any code you’re squeezing for maximum performance. So maybe this is one to ponder for a while, and keep up my sleeve for appropriate situations.

Advertisements

Actions

Information

One response

2 08 2013
Some more testing against different JDKs… | Closing Braces

[…] For details of the other JDKs that were already included in the testing, further background information and previous findings, refer to the earlier post Some specific issues from ObMimic portability testing. […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s




%d bloggers like this: