I’ve been spoilt. Thoroughly spoilt. I’ve spent the last few years working on things that can be tested. Sure, it’s a lot of work writing good tests, but once they’re in place testing is a breeze. Run the script (or push the button), the tests run, and either it’s ok or there’s a list of problems to investigate and fix. New version? Made a change? Refactored some code? New environment? Need to prove something still works? Just run the tests again.
But now the day of reckoning has come. I’m doing some part-time work for a company that provides web-sites based on their own CMS product, but with lots of web-page design and bespoke customisation for each client. It’s very good at what it does, but testing each individual site is just a massive manual exercise. No automation at all.
It seems like a huge drain on resources, and unreliable and unrepeatable as well. Each new client or redesign leads to a fresh set of testing, and it all relies on you knowing what to test and what to look for. It’s done on a relatively undocumented and informal basis. Unsurprisingly, they seem to have lots of “regressions”, bugs they have to deal with over and over again.
What really bothers me is that I’m struggling to see any solution that would be more cost-effective.
The bulk of the testing is checking that web pages display exactly as intended, with correct layout, styles, positioning etc – compared to some mock-up pages provided by the graphic designers (which are substantial, but by no means exhaustive), and relying heavily on human judgement (e.g. “surely there’s supposed to be whitespace between those two elements when they are side-by-side… and shouldn’t that line be the same thickness as that one…”).
That sounds simple, but the pages combine variable numbers of different types of content, with lots of different options, combinations of layouts, complex criteria that decide each page’s content, many different optional elements within items, special formatting for particular elements when used in particular combinations, elements where the client can supply arbitrary HTML. The list goes on and on – the permutations are as near to infinite as makes no odds.
The end result is that even when the underlying CMS functionality is taken as being stable and already tested (which is never entirely true, of course), there’s an absolutely huge amount of visual checking of web-pages to be done.
Inspection of the CSS and HTML doesn’t even begin to give you any idea of whether it has the intended effect in all situations and no nasty surprises, let alone whether it works in different browsers and window sizes. Exact pixel-by-pixel comparison of each page’s image against an expected result might work, but there would be an absolutely enourmous number of expected results to set up, and it would all be very fragile (e.g. even the smallest change to spacing or to an element that appears everywhere would require all the “expected” images to be re-done).
So there seems little alternative other than to work through the various combinations and visually check each resulting web-page against the mock-ups (with exact comparison of individual elements where this seems appropriate). And do that for all the different combinations of elements and other features. All whilst figuring out which part of which mock-up page to look at for how each part of each page should look.
It might be ok if this was for a single web-site, or a CMS on its own, or a CMS with some simple “skins” with a limited set of variations. Then you could at least consider investing in an overall set of tests, maybe with pixel-exact checking of bitmaps against the design. It’d be a lot of work, and need a lot of maintenance, but if there was just one such set of tests for your entire business (and especially if you could adjust the “design” process to take testing into account), it might be worthwhile. At worst you could at least justify spending time on producing a good test plan for doing the manual testing.
But it isn’t like that. Instead, the styling is different for each client/web-site, and often with other bespoke features just for that particular client. It’s all basically a “one-off” exercise for each client. Having a few people informally bash away at a test site for a few days is far less work than trying to somehow automate this, or even develop and document a solid test plan – much of which would then need substantial revision for the next client.
I have a few ideas to help improve their overall approach, and I’m well aware of tools that could automate testing of the “functional” aspects of their CMS. But it’s come as a bit of a shock to find that not only do I not know of any tools to automate the “visual” aspects of the tests, but I can’t even envisage tools that could automate this (at least, not without requiring even more work than manual testing).
Does anyone else face this problem? Surely it’s a fairly common requirement to want to check the appearance of a set of web-pages against their design? Is there an obvious solution that I’m missing? Any solution at all? Or is this really something where there’s no alternative to homo sapien and the Mk1 eyeball?
In the meantime I’d better get on, I have a huge set of web pages to examine…