In this study we are comparing the findings between two competing user testing methods (and technologies); eye tracking (Tobii T100) and conventional user testing software (Silverback).

We wanted to test an equal amount of users using each method but due to logistical issues this did not happen (we tested 7 users on the Tobii T100 with 4 users on Silverback).

The website we tested was OfficeCavalry.com – produced by Cyber-Duck (also the authors of this user testing ‘test’). The demographics were undergraduate students of multimedia and interactive media from the University of East London.

Tasks 1a - Register as a Freelancer

The users were asked to register on OfficeCavalry.com (as freelancers) using their own credentials or fake credentials*. We measured the time it took users to finish the registration process from start to finish. All users were able to complete the registration within an average of 2 minutes and 52 seconds.

*We let the user choose how they wanted to register to make them feel as comfortable as possible

Figure 1.1 – As shown in this picture the most eye catching areas were the rotating illustration on the main banner, the register CTA 02 on the top right corner, and the search widget on the main banner.

Using both user testing techniques we were able to state that the CTA 01 on the main banner wasn’t performing with only one user (out of 11) clicking on it. By analysing the gaze density via the Heat map within the Tobii software we were able to determine that the users were not focussing on the CTA.

We wanted more evidence, so created an AOI (Area of Interest) within Tobii studio. The AOI proved that only 4 out of 7 users looked at the CTA with an average of 4 fixations and an average time of 0.30 seconds per fixation. This confirms that users were not gazing at the CTA.

Tasks 1b – Registration form

Figure 1.2 – As we can see, the most eye catching areas were the ‘labels’ and ‘fields’. Using eye tracking gaze data and AOI functions we could see very little focus on the ‘look up’ button possibly due to a design issue.

Using both user testing techniques, we noticed that 4 out of 11 users did not click the ‘look up’ button next to the postcode (zip) input field. Using the Heat map facility via Tobii studio we determined that many of the users did not click on the ‘look up’ button because they were not gazing at it enough – this was quite evident on the heat map.

After creating an AOI (via Tobii studio) we proved that all the eye tracking users, besides one, gazed at the ‘look up’ button (Button) with a mean of 6 fixations (average time of 0.46 seconds per fixation) during the test, confirming that the postcode ‘look up’ button was grabbing some user attention (but users were not clicking on it).

Another issue which affected the user completing the form (relating to the address look up) was a (positive) tick indicator that appeared as soon as the user clicked within the post code input box regardless of whether they entered a post code or not. As a solution to these two issues we would create a bolder coloured button (with a higher contrast ratio) and fix the address form validation issue with the positive tick indicator appearing at the right time (as opposed to prematurely).

Thanks to eye tracking technology, we were able to see where users were gazing and discover that the users that clicked on the ‘look up’ (label) button didn’t look at it enough to take it seriously. We can therefore confirm that the users that did click the post code ‘look up’ button almost definitely did so because they were in-fact familiar with this type of functionality.

The registration speed should be improved. The current average is 2 minutes and 30 seconds with the fastest user completing the registration within 1min and 40sec. As a result of the user testing we propose shifting the address capture to post registration.

Task 2- Profile yourself on CreativeCavalry (both ET and SB)

On this task, users were asked to login using a pre-defined username and password. After ‘logging in’ users were asked to create a professional profile on CreativeCavalry including selecting their profession, skills and creating a CV using the CV builder tool.

As we were giving users pre-defined credentials (to speed up the task for the purpose of this test) they did not land on the welcome page like a new user would normally do on their first login. We should therefore be more sympathetic towards users in this test as a new or ‘real’ user would go through an introductory welcome page and be more familiar with the user experience and interfaces.

Figure 2.1 – This is the My Profile ‘builder’ where the user builds up their professional details. As shown in this picture the most eye catching areas were field labels and input fields. 2 users out of 7 did not look at the save button (Button).

3 out of 11 users found it difficult to locate the ‘save’ button when building their profile and going from tab to tab. 9 out of 11 users clicked on a tab rather than the save button, losing the data they entered before (due to not having adequate functionality to cater for this scenario). Whilst producing the website we assumed that users would click save but thanks to (both techniques of) user testing, we found that this was not the case.

Thanks to the eye tracking technology and after creating an AOI (via Tobii studio) for the Save button (Button) we determined that the 5 out of 7 (eye tracking) users who did not click on the ‘save’ button (Button) had nil fixations on that particular button. We therefore recommended enabling the ‘saving’ of data when user is moving from tab to tab.

Both user testing techniques allowed us to see that users did not seem to understand the flow they had to follow in order to complete their profile. The tab system on the ‘my profile’ section should probably have numbers like a progress bar and be treated differently for first time logins.

Figure 2.2 – This is the screenshot that shows that the user was not looking at the end date. Without eye tracking it would be hard to analyse this.

As a result of seeing the gaze data using eye tracking technology we noticed that 4 users were confused about how to enter date information for education details (namely the end date, this could be due to the fact that they were still studying and this field was not relevant). All 4 users entered the ‘start date’ for education details and professional qualifications but did not enter the ‘end date’.

Figure 2.3 – Similar to figure 2.2 (where the user enters educational details), this screenshot shows issues with the dates upon entering professional qualifications.

This resulted in the users getting frustrated after receiving system notifications that did not allow them to move forward.

Figure 2.4 – This screen is highlighting the error message which is not in situ (i.e. not next to the date field.

We also recommend making notification messages more prominent as users are not inputting all date information.

Summary

From our research we were able to spot the same amount of usability issues but thanks to Tobii studio we could analyse the results faster and in more detail.

Eye tracking is a much more scientific way to approach user testing as it allows practitioners to see where the user gaze is focused. This gives detailed insights into unconscious user behaviour ‘in between the clicks’.

Tobii software provides practitioners with meaningful data such as number of fixations and durations, AOI and heat maps. On the other hand eye tracking user testing requires a longer setup time, more equipment and configurations. Eye tracking also tends to achieve best results in fairly formal settings. These factors may make the eye tracking participant feel slightly less comfortable than using software like SilverBack on a conventional laptop in the same location.