Through careful interface design, coupled with the use of animations and audio, we are now able to reliably assess candidates with lower levels of literacy
Instead of simply getting the answer right or wrong, we can measure how close or far the candidate was from the perfect answer
Online assessment is a phenomenon that has become central to recruitment. Computer-based testing has existed for more than 20 years, but the ever-expanding Internet has pushed psychometric testing out of the interviewers’ offices and into the candidates’ homes.
The first generation of online testing consisted of paper-and-pencil tests such as verbal reasoning. This format of text-based tests was easy to implement at a time when broadband connectivity was limited to business and educational institutions.
The second generation witnessed the introduction of graphics into online testing, allowing for charts, tables and diagrams of numerical and logical reasoning tests to be displayed with fidelity very close to traditional printed tests.
These first two generations spanned from the late 1990s till as late as 2005. Now, well into a third generation, online testing is undergoing a rapid and dramatic transformation—one where tests don’t have to look like tests and candidates take a much more active role in their own assessment.
So, how is this third generation of testing used in specific industries? We can take hospitality and tourism as an example. If we are recruiting housekeeping staff, one of the criteria we want to assess is memory. We want to know if they can remember the layout of a room’s furniture so that they can return objects to their appropriate location. In this type of test, the screen will show a room for example, the candidate must remember the location of 15 items after viewing them for 20 seconds. He/she must place the items in the appropriate place by dragging and dropping them. So now, instead of simply getting the answer right or wrong, we can measure how close or far the candidate was from the perfect answer. Not only does this reduce the chance that the candidate gets the answer right just by luck, it also allows for a much higher resolution of measurement, and therefore more differentiation between candidates.
In addition, test authors can now design the test interface from scratch instead of having to “shoehorn” new tests into old formats. This means that intuitive interfaces can be designed to reduce the risk of misinterpreting the instructions or looking at the wrong screen location. In another example, candidates are shown a situation (one where a hair dryer has been put in a hotel bathroom sink full of water while still plugged in) and must choose from four possible options. A narrator also reads any text that appears on screen. When choosing the correct option, the candidate sees small animations of each of the possible actions. This has the effect of minimising the literacy level that the candidate must have to understand the instructions and complete the test.
That last point is significant. Through careful interface design, coupled with the use of animations and audio, we are now able to reliably assess candidates with lower levels of literacy. Many jobs will attract candidates with lower-than-average literacy, so this style of assessment may prove extremely useful.
The increasing use of multimedia is having a big impact on online Situational Judgement Tests (SJTs). An SJT consists of a number of questions that describe a hypothetical scenario and ask the candidate to indicate what he/she thinks would be the best resolution. The effectiveness of SJT performance predicting job performance has long been established. However, in their usual written format, candidates with lower-than-average literacy levels may perform poorly because their comprehension of the scenario, and of the possible responses, affects their results.
Research in the 1990s indicated that replacing text-based scenarios with video and audio scenarios had a positive effect on minimising some of these literacy effects. Of course, the cost of producing a video SJT far exceeded the cost of producing a text-based SJT, and so video SJTs have not been commonplace in the past.
However, as production costs have decreased, video and animated SJTs have started to become more common. With the advent of low-cost animation software, animated SJTs can be created even more economically than their video counterparts.
Finally, we can measure skills specific to certain roles (hospitality in our example) with a higher degree of accuracy, because we are not having to assume a certain level of literacy from the candidate. And because these tests are online, then we can put tens or tens of thousands of candidates through them with no additional work or effort on the part of recruiters.
So what is the next step for online testing? We are going to be seeing more ‘immersive assessments’. These are assessments that are set in a virtual world (similar to Second Life) when the user can walk around, observe, explore and interact in many of the same ways they can in real life. Immersive assessment has been used for many years in military training, for example. Now, it is a viable platform to assess in commercial settings. So expect to see more realistic simulations of situations such as customer service, sales, complaint handling and so on. In fact, any interaction which your employees may face in the real world can now be replicated – and their responses measured – on an immersive assessment platform.
About the Author: Ben Hawkes is the head of simulation development at Kenexa and specialises in the use of technology to deliver efficient and accurate assessment solutions. With more than 10 years of assessment experience in the U.S. and UK, Mr. Hawkes has designed and delivered assessments for organisations such as Merrill Lynch, HMRC, E.ON, Royal Mail, and the Foreign and Commonwealth Office. He also serves as a trainer for job analysis, competency modeling, interviewing, assessment centres, performance management and psychometric testing. Before joining Kenexa, Mr. Hawkes ran his own consultancy practice for six years, specialising in psychometric assessment, training and development and workplace surveys for telecom, banking, marketing, IT, manufacturing and public sector clients.