Background

Since the beginning of the year I have been working on developing a new website constructed using real user research data and user testing data for Nellie’s Nursery, a local nursery. The progress has been documented in weekly reflective journals and production diaries, starting on January 8th 2018. Recently I have been developing three early high-fidelity evolutionary prototypes for the first stage of user testing. These early prototypes give an idea of how I have built a very basic version of the website using a small amount of user data that has already been collected and show the direction in which further development of the website could go. The idea of testing is to find bugs, find out how usable the prototypes are, what users like about the way the site is headed, what they don’t like, how easy it is for them to complete simple tasks and give them a chance to voice their opinions before any further any great amount of time is invested in further development.

Prerequisite reading

For more background information, please read the following previous posts:

• For information on the user research that has been used to construct these prototypes, please read the Reflective Journal from 5th-9th February 2018.
• For information on the prototyping methodology, strategy and explanation, please read the Reflective Journal from 12th-16th February 2018.

Prototyping

Methodology

The first round of prototyping was carried out on Thursday 15th February 2018 in the UX Lab located at Norwich University of the Arts. Five people completed the testing. The aim of the prototyping session was to test three different prototypes in an A-B-C test and get each user to carry out the same (or very similar) tasks on each prototype, specifically:

• (A) The user must firstly find the session times and tell me what they are.
• (B) The user advance the slides on the slideshow.
• (C) The user must navigate to the ‘Groups’ page.
• (D) The user must navigate to the:
  • Desktop: ‘Jolly Giraffes’ section.
  • Mobile Prototype 1: ‘Little Lions’ section.
  • Mobile Prototype 2: ‘Mini Monkeys’ section.
• (E) The user must tell me the age range of the children in the group they had to navigate to and the rooms that that group use.
• (F) The user must navigate to the ‘Day-To-Day’ page and tell me some information about either the Courtyard or the Nature Garden. They cannot repeat the same information they told me last time,
• (G) To complete this stage of the testing, the user must navigate back to the home page.

These tests collectively evaluate:

• How easy information is to find, this is verified by getting them to tell me some information that’s displayed on the page that they have visited. I won’t prompt them to tell me some information, they’ll have to tell me when they’ve found some (to prompt this, instructions will be worded like ‘navigate to the
• How easy the site is to use – did they manage to find the pages and make the slideshow advance?
• This data is backed up using quantitative data by using eye-tracking. Heat and gaze maps from the eye-tracking software can be used to evaluate where the user is looking on the screen, meaning I can once again evaluate if they are looking in the right places, clicking on the right things and so on in order to build a picture of how easy the site is to use in terms of UX and information architecture.

After the first stage of the test has been completed, I gave users an opportunity to answer these open questions which allowed me to collect some qualitative data to allow my testers to elaborate on certain points.

• Information architecture questions
  • What kind of information would you expect to find on a nursery website?
  • Going by the buttons alone, would you say that this site has all of the information that you’d expect to find on a nursery website?
  • Was there anything distracting or misleading on the pages?
• UX questions
  • Did you find finding information on the page easy? Was the information clearly laid out and easy to read and find?
  • Generally, did you prefer navigation on the desktop or the mobile site?
  • Was the navigation more logical to use on prototype 1 or prototype 2?
  • Which prototype was it easier to find the pages on?
  • Were the icons on prototype 1 logical and easy to work out?
  • When using the handset, did you find it more comfortable to navigate on prototype 1 or prototype 2?

Data was collected using the Tobii eye tracker, a Logitech webcam recording the user using the prototype on the device and my camera was used to record a video of each interview after the test.

The test was originally going to be done on the desktop PC, followed by doing the test on a smartphone (I had a Samsung Galaxy S7 in mind which I chose because we have them at the university for testing purposes and because it is a modern smartphone, representative of the type of device that many people own. The S7 also has a bright display that will display well in a video). However, upon testing the rig before testing on the day of testing I learned two things:

• Firstly, there were quite a few complications involved with doing the test on a desktop and then on a mobile device. The eye tracker would need to be moved, several devices unplugged and then several changes made in the software preferences of the recording software. I could probably have remembered the order in which to do things, but I was aware that I was going to be limited with time since I only had about 20 minutes or so with each user and having tested 6 other people’s prototypes before my own, I realised that my test was already quite long-winded. I wanted to make use of the time I would have available to me collecting data, not unplugging cables and changing software settings.
• Secondly, I tried the eye-tracking on my website running on my Nokia Lumia 930. I soon realised that doing eye tracking on a device that has a 5″ display is difficult because when looking at the gaze map the dots were all on top of each other. It was almost impossible to get any meaningful data from it. It just looked like the whole screen was being looked at the whole time, possibly because the webcam recording the user was quite high up above the phone. In its day at 5″ this was a large smartphone, but now it’s about average size. Even still, this large(ish) smartphone is still larger than an iPhone 6, 7, 8 and quite a few other popular phones available these days. It’s about the same size as the S7 I had planned to do the test on.

In the end I did the test on an iPad. This had its advantages and disadvantages. The advantages were that I could do the test on one device and it would be able to emulate the desktop site and both mobile ones. In landscape orientation, the iPad displays a site that looks like the desktop ones and in portrait orientation it displays the mobile ones. The other benefits were that by using the iPad I was able to save time by not having to worry about changing the hardware or software setup for the test and the iPad screen being 9.7″ was big enough to get a better gaze map. With the iPad being bigger than a phone the webcam view also looks better and is clearer too.

The disadvantage of using an iPad was that my desktop results aren’t truly representative now because the desktop test questions were written to test usability using a mouse and keyboard and testing the site on a large display, but at the end of the day the user was able to see how the site would look on a desktop device like a PC or a laptop. The portrait mode puts the site into mobile mode and works well, but by using an iPad I wasn’t able to get the user to try and pick up the handset and compare the ergonomics of the different prototypes on a smartphone and also see a big difference in scaling between the desktop and mobile sites so that they could definitely see that the prototype is responsive and looks differently on smaller screens. To get around the ergonomics issue I gave some of my users my phone in the interview and asked them if they felt that the prototype with the static menu bar was easier to use in one hand than the prototype with the hamburger menu.

When doing testing like this on quite a small scale, you have to decide what’s more important. For me, timing and setup was an issue and the iPad gave a good enough representation of the page running on a desktop and mobile device, so it worked OK.

The iPad I used was a near-enough brand new iPad 5 which the university have on hand for testing purposes (like the Samsung Galaxy S7s and iPhone 7s). The site looked and worked in the same way as it did on the older iPad 4 that I tried it on last week.

Five people tested my prototype, all of whom were BSc students. To try and broaden my test base I later got my parents to complete the tasks on February 18th at home on the iPad 4 that I previously tested the prototypes on. The tests were conducted in exactly the same way with the same questions asked and the same tasks set for them to complete, but the difference is that there was of course no eye-tracking since I do not have eye-tracking hardware at home. Having evaluated some of the test data before these additional tests were run, I had decided that in this test at least observation was more important for determining the usability of the prototype than analysing the eye-tracking, so instead I used my Nikon D500 XT with a Rode VideoMic attached to record them using the prototype on the iPad which I could later watch and analyse.

I was keen to get my parents to try the prototype testing because one mistake I made when running the prototyping for Stellardrive was that only 5 people tested it and they were all very similar people: all with little-to-none driving experience, all a similar age and all interested in technology being BSc students. I appreciate that this is a small-scale project and I have limited time and people to test my prototype(s), but to get more interesting data and better identify patterns I wanted as many people to complete the testing as possible. Some sources will say that 5 testers will generally find 80% of the faults, which is possibly true, but who will find the remaining 20%? With the car infotainment system prototype that I ran testing for back in November 2017 it wasn’t such a massive problem that only 5 people tested it because although these 5 people had very little driving experience and were also all BSc students, they hadn’t really been closely following the development of my work and although they were also running prototypes for their work, they weren’t making car infotainment system prototypes, so my when they went to test my prototype it was genuinely the first time that they had seen it. This time the prototype is a website and most of my close peers knew that I was working on three prototypes to test and that one would be for a desktop site, one would be a mobile site with a fixed menu bar at the bottom of the page and one would be a mobile site with a hamburger menu. These people also all study the same thing as me so perhaps have the benefit of some ‘insider knowledge’ of how I expected the prototype to be used. My parents, whilst interested in my studies, don’t know all of the ins-and-outs in the same way that my peers might.

Data Analysis

Data was collected in several forms:

• Gaze maps from the Tobii eye-tracking software (only for tests completed on February 15th) – heatmaps could also be exported but were not because I personally feel that gaze maps are more useful. They show the order in which the user looks at elements as well as where they were looking and the larger the spots, the longer they were looking for.
• Observation videos from the webcam on the mobile testing rig (for tests completed on February 15th) or observation videos from the Nikon D500 (for tests completed on February 18th).
• Interview videos in which the testers express their opinions.

Unlike the testing for the Stellardrive infotainment system, each of the seven people who tested my website generally had very similar comments to make about the website and all ran into similar pain points. The data for each person who tested my prototype is presented below.

Celestin Jacobs

Celestin studies BSc Games Development and previously tested the Stellardrive prototype. He was the first person to test my Nellie’s Nursery prototypes.

Observation 

Unfortunately there is no audio for the testing of the desktop prototype or the first mobile prototype, but luckily the audio is restored in the testing video for the second mobile prototype. See the testing footage below.

The observation video gives the following insights to how Celestin used the prototype to complete the tasks set. Here are some key observations from the testing.

Testing – Desktop

• Tried to find session times initially by going onto the Day-To-Day page and spent a long time looking for them on there, then going onto the groups page and found them there. Read them all fine.
• Instinctively advanced the slideshow by swiping on it.
• Found the Jolly Giraffes information by scrolling down the page until he found the appropriate section on the page. Went back to the top of the page by pressing the up icon. When asked to do the test a second time (there was an issue with my prototype that prevented the test from doing what it should) he did click on a class icon to take him to the right section and it did work.
• Found the Day-To-Day page easily and could read the information about The Courtyard.
• No problem returning to home page.

Testing – Mobile 1

• Instinctively looked for the session times on the home page, found them quickly and read them with ease.
• Instinctively swiped the slides again.
• When asked to go onto the groups page he went onto the Day-To-Day page at first, then tapped on the groups icon, didn’t see he had tapped on it, then tapped on the Meet The Team icon, then tapped back onto the groups icon. He was tapping the buttons until he found the correct page.
• Tapped on the Little Lions icon to navigate to that section and found the information fine.
• When asked to go to the Day-To-Day page he accidentally tapped on the icon to go to the Meet The Team page again. He found the information about The Nature Garden though.
• No problem returning to home page.
• When asked what each of the buttons at the bottom do, he understood home and contact, but the others he was unsure of. He really didn’t understand the Meet The Team icon.

Testing – Mobile 2

• Had some trouble activating hamburger menu, originally went onto Day-To-Day page, then went onto the home page, had to ask if it was the home page, then found them there.
• Swiped the slideshow.
• Navigated to the groups page with no problem.
• Found the age of the children in Mini Monkeys and navigated to the section by tapping on the icon.
• Found the Day-To-Day page easily. Easily read the information about The Courtyard.
• No problem returning to home page.

Gaze Maps

The video below shows his gaze map when testing the prototypes. Unfortunately we could not get the eye-tracking system to correctly calibrate Celestin’s eyes, but being a similar height to me I used my profile and it generally recorded the data with some level of accuracy. However it is clear to see that the profiles were not fully compatible and the system detected his eyes looking above the iPad a lot during the desktop testing, but when testing the mobile sites it was generally more accurate. Unfortunately the Tobii software does not export audio when exporting gaze maps.

Trying to ignore the calibration issues, the gaze maps suggest that generally Celestin was looking in the correct places on the prototype to complete the tasks. However, when this gaze map data is combined with the observation video and his actions are observed in more detail, although he was looking in the correct places on the prototype (generally) for data, it was taking him a while to find the correct location of the data. For example when trying to find the groups page on the first mobile prototype he knew to look for the button to take him to that page on the menu bar at the bottom of the page, but what he didn’t know immediately was which button to press to take him to that page.

Interview

His interview video is below.

Key points from the interview with Celestin:

• From a nursery website he’d expect to find out: what my kids are up to, where it is, when I can go, activities, news.
• Website has all info on it apart from where it is.
• Nothing misleading or distracting. Some things could be clearer – didn’t understand the staff button on the mobile prototype 1. Thought labels underneath the icons would have helped.
• Information very easy to find.
• Favourite navigation was the hamburger on Mobile Prototype 2. Felt it was the quickest.
• Hamburger presented clear options and felt natural. Used to using hamburgers on other websites and other apps. Mentioned that swiping could be used for navigation like on Twitter app (swipe right to reveal hamburger menu).
• Didn’t think the fixed menu was easier to ‘thumb’ – buttons too small, if bigger it would be better. (thumbs pressing wrong buttons, then suggested swiping might be even easier than thumbing buttons).
• Liked the use of Lorem Ipsum.

Kieran Adams

Kieran was next to test my prototype and also studies BSc Games Development. He didn’t test the Stellardrive prototype, so this is the first time he has ever tested a prototype for me.

Observation

There is audio for all three parts of the observation video.

The observation video gives the following insights to how Kieran used the prototype to complete the tasks set. Here are some key observations from the testing.

Testing – Desktop

• Found session times on the home page.
• Used the previous and next buttons to advance the slides on the home page.
• Found the information about Jolly Giraffes and instinctively clicked on the class icon to go to the section and the arrow button to go back to the top of the page.
• Found Day-To-Day page perfectly and information on The Courtyard was found and read easily.
• Could go back to the home page easily.

Testing – Mobile 1

• Found session times on the home page.
• Advanced slides using the buttons. When asked if there was another way he tried to tap the edges of the slideshow and then he swiped.
• When asked which page each button on the menu takes him, his answers were:
  • 1 – Home
  • 2 – Might be about
  • 3 – Didn’t know
  • 4 – Could be something like groups
  • 5 – Parent information
  • 6 – Contact
• Tapped on the Mini Monkeys icon to navigate to that section and found the information fine. Pressed arrow to go back to the top.
• Struggled to find the Day-To-Day page. Started scrolling up and down the Groups page he was on, then went onto the home page, saw it wasn’t where he needed to be and then tapped on the icon next to the home button and landed on the Day-To-Day page. Found the information about the Nature Garden easily.
• Navigated back to the home page easily.

Testing – Mobile 2

• Found the session times on the home page.
• Swiped the slideshow to advanced frames, then used the buttons beneath.
• Found the groups page easily and quickly.
• Found the age of the children in the Little Lions group but I accidentally instructed him to tap on it.
• Found the Day-To-Day page easily and quickly. Found information about the courtyard quickly and easily.
• When asked if there was a way to see the page you were currently on without scrolling to the top he said the hamburger menu told you.
• No problem navigating back to the home page.

Following Celestin’s confusion with the menu buttons in Mobile Prototype 1, Kieran’s test was the first one to feature the question ‘what do you think each of these menu buttons do?’ I realised when Celestin was completing his test that I knew he didn’t understand what at least two of the six buttons did, but I didn’t know what he thought the others were for. From here on I decided to ask the candidate what they thought each button did in order to understand which icons were obvious to the candidate and which ones were not. Noticing that Celestin seemed to not always know which page he was on on the prototype too, I started asking candidates at this point onwards if there was a way to find out which page they were on and I often asked this question during the testing of Mobile Prototype 2 which featured the hamburger menu. I should really have asked him on both prototypes in hindsight but I felt it was less obvious on prototype 2 so wanted to see if candidates could recognise that the button of the current page was highlighted orange.

Gaze Maps

The eye-tracking system was able to calibrate Kieran’s eyes and successfully create a profile for him but the calibration is still not perfect. The lab staff did inform me that the eye-tracking works much better on the desktop computer than it does on the mobile device testing rig, so this could be partly it. I think it’s because the eye sensor is located on the bezel of the monitor for the desktop PC when you are testing on, but on the mobile device testing rig it has to sit underneath the device so sometimes it can be harder to calibrate as to initially set it up you need to stare at it with your head almost pointing towards the floor and then when you are actually using the device your eyes are not being picked up the tracker because you are not looking directly at it, like you are when using the desktop computer.

The gaze map from Kieran’s test is not the easiest to interpret and overall doesn’t really provide the kind of insight that I had hoped for. All it really shows is that he is looking at the iPad and generally in the correct place on the iPad, but again this does not mean that he did not experience pain points or sometimes take a while to find the pages that he was looking for. This only becomes evident when the gaze maps are viewed in conjunction with the observation footage.

Interview

His interview video is below:

Key points from the interview with Kieran:

• From a nursery website he’d expect to find out: what they do, what the children do, how much it is, where it is.
• All information there apart from location button.
• Nothing that confusing or misleading but on the first mobile prototype got confused finding the ‘groups’ icon – only found it because the buttons have the same hierarchy as the desktop site which was tested beforehand. Didn’t know that the giraffe meant ‘groups’. A new user wouldn’t know that the giraffe icon related to the name of the groups in the nursery. There was an icon which showed a group of people which he thought was groups – it was actually ‘staff’.
• Thought the menu bar with the icons at the bottom looked quite cool when he first saw it. If children were to use this website they might enjoy that more than the hamburger. Icons are more unique and playful.
• When given the phone to play with the bottom menu on, he noted it was more of a stretch to go to the hamburger menu. However, he mentioned that a lot of apps generally have tabs and buttons at the top of the interface too. Fixed menu bar at the bottom is quite easy to reach when you have just one hand. On a tablet it’s not so much of a consideration because generally you use tablets with two hands.
• Pencil for the day-to-day page wasn’t obvious, he kind of got it. Took a while to figure out what it was.
• Liked the A-B-C nature of the test itself.
• Liked that the buttons on the bottom were more unique than a hamburger menu.

George Britton

George also studies BSc Games Development and like Kieran has also never tested a prototype for me before.

Observation

See George’s observation video below:

The observation video gives the following insights to how George used the prototype to complete the tasks set. Here are some key observations from the testing.

Testing – Desktop

• Tried to find session times initially by going onto the Day-To-Day page and spent a long time looking for them on there, then going onto the groups page and found them there. Read them all fine.
• Used the buttons to advance the slideshow.
• Navigated to the Groups page with ease and found the age of the children in the Jolly Giraffes class easily. Clicked on the class icon to go to the page and started to scroll to go back to the top of the page, but then saw the up icon and used the upwards arrow button to go back to the top of the page.
• Found the Day-To-Day page and read information about The Courtyard with no problem.
• (Forgot to ask him to go back to the home page).

Testing – Mobile 1

• Started looking on the home page, but before he got to that bit of the page went onto the Day-To-Day page, then onto the Groups page and found the information there.
• Used the previous and next buttons to advance slides again, when asked if there was another way he instinctively swiped left and right.
• Navigated to the groups page fine. When asked if he clicked on it because it was obvious to him that this was the groups page or because he had clicked on it before and it had taken him to that page, he said he remembered the ‘Jolly Giraffes’ so he clicked on it because there was a giraffe on the button.
• Tapped on the Little Lions icon to navigate to that section and found the information fine.
• Found the Day-To-Day page easily and information on the Nature Garden was found and read easily.
• Returned to the home page with no problem.
• When asked which page each button on the menu takes him, his answers were:
  • 1 – Home
  • 2 – Day-To-Day
  • 3 – Groups
  • 4 – About us
  • 5 – Parents
  • 6 – Contact

Testing – Mobile 2

• Found the session times on the groups page quickly and almost instinctively, read them fine.
• Swiped slideshow and pressed the buttons.
• Easy and quick to navigate to groups page.
• Found the age of Mini Monkeys and navigated to section with no problems.
• Navigated to Day-To-Day page easily and found and read information about The Courtyard easily.
• Easy to see which page you are on because of the orange.
• Navigated back to home page flawlessly.

Gaze Maps

Kieran’s profile was used for George’s gaze maps. The calibration for the desktop testing is completely off – you need to imagine that the spots are actually over the iPad in order to get an accurate representation of how the data should look. On the mobile prototypes the calibration is better. The large spots on the mobile prototype testing reveal that George was generally looking in the right place for links and information, however he sometimes spent a little while looking at them before he thought to tap on them. This could indicate that either he was a bit nervous during testing and perhaps more hesitant than he would have liked to have been when using the prototype, maybe he was waiting for further instruction or some kind of guidance before tapping buttons or maybe he genuinely had to think twice about what he was going to do which would indicate UX problems. The only way to find out for sure was to interview him which is what I did and why I found that collecting all of these different bits of data was imperative to be able to draw final conclusions about the testing.

Interview

George’s interview is below:

Key points from the interview with George:

• From a nursery website he’d expect to find out: session times, location, what their values are, child-raising strategies, how they deal with children.
• Website seems to have all the info on it, but didn’t go through all the pages and didn’t see any contact information, but assumed that this would be on the contact page.
• Nothing distracting or misleading but it wasn’t clear that you could swipe through the slides in the slideshow.
• Finding information on the pages was easy. Easy to tell what you were looking at.
• Desktop navigation easiest. Buttons with words easy, didn’t have to click a menu, see where you were without clicking a menu.
• Preferred the first mobile prototype. Liked the non-worded buttons on the mobile site. Prefers a menu that is outside of a menu button. Always accessible.
• Imagined it being easy to thumb buttons at the bottom of the phone.
• Icons logical on first prototype. Didn’t like pencil icon for ‘day-to-day’ but not obvious, suggested a calendar instead. Did understand the icon once was on the page.
• Could see it being a legitimate nursery website.

Naomi Winter

Naomi studies BSc User Experience Design like myself and was one of the testers of the Stellardrive prototype.

Observation

See Naomi’s observation video below:

The observation video gives the following insights to how Naomi used the prototype to complete the tasks set. Here are some key observations from the testing.

Testing – Desktop

• Found session times on the home page but clearly had to think for a few seconds before starting to scroll down the page.
• Instinctively swiped the slideshow to advance the frame. Went back to the top of the home page by pressing the menu button again.
• Read the age of the children in the Jolly Giraffes class easily, but didn’t click on the icon to go down to the section – scrolled instead. Went back to the top of the page by tapping on the arrow though.
• No problem navigating to the day-to-day page and reading the information about the courtyard.
• No problem navigating back to the home page.

Testing – Mobile 1

• Found the session times on the home page, had to scroll up and down the home page a bit to find them though.
• Swiped the slideshow to advance the frames.
• At first tapped on the Meet The Team icon to find the Groups page. Then tapped on the Parents page button. Then scrolled down the Home page. Then pressed the Home button. Then tapped on the Day-To-Day page button. Then scrolled down that page a lot. Then by process of elimination tapped on the correct button.
• When asked which page each button on the menu takes her, her answers were:
  • ‘I have absolutely no idea […] quite a tough menu system to decipher.’
  • 1 – Home
  • 2 – ‘That’s very confusing. That doesn’t scream Day-To-Day to make either – it looks more like a customisation option which is weird because you’re on a website trying to find stuff.’
  • 3 – ‘That did not indicate groups at all’ then realised that it was a giraffe on the button because there is a group called ‘Jolly Giraffes’, but then pointed out that there were monkeys and lions as well.
  • 4 – ‘I thought that was groups because it has a group of people […] this could be a community page? Can’t remember what it was. Something like a forum I immediately thought.’
  • 5 – Parent information
  • 6 – Contact
• No problem navigating back to the home page.

Testing – Mobile 2

• Session times found on the home page, but interestingly played around a tiny bit with the slideshow first.
• Found the Groups page easily but had to tap the hamburger menu icon a couple of times to make it work. Commented that the navigation was ‘a lot better’.
• Found the age of the children in the Mini Monkeys class fine and tapped on the icon to go to the section, but mentioned that the page navigated slightly too far with some information going underneath the header.
• When asked if it was easy to see which page you are currently on, she said no but when asked to check the hamburger menu she said it was highlighted, although it’d be easier if the page that you are currently on appears in the header area, or a small icon appears at the bottom of the page that depicts it. When asked if it was more obvious on mobile prototype 1 she said yes. Suggested that navigation from both prototypes somehow needs to be ‘blended’.

Gaze Maps

The eye-tracking system could not create a profile for Naomi so instead she used mine. Unfortunately she is a little shorter than I am so the calibration was pretty poor and again suffers from the issue where the spots appear above the device when testing the desktop prototype, but the mobile prototype gaze maps look OK, if a little off-calibrated. Like gaze maps from the other testers, they don’t really show a lot of useful information, only really that it appears that she was looking in similar places on the device as the other testers were and that where she was looking seemed like a fairly concentrated area – meaning that she tended to be looking quite closely at the centre of the device and not so much at the edges. Perhaps this indicate that this is where she expected to find the content or that the content of the page appears to be in the centre of the page. This gives an insight to content organisation on my prototype. It is also possible that there are maybe some distracting elements that caught her attention – the only way to find out if the elements were distracting would be to interview her and get her opinion on it which again is why interviews are used to collect qualitative data as well as using something like eye-tracking to collect quantitative data.

Interview

Naomi’s interview is below. Just like when giving feedback for the Stellardrive prototype, I felt that Naomi gave more detail in her answers and provided more insight than most of my other testing candidates did.

Key points from Naomi’s interview:

• From a nursery website she’d expect to find out: opening times, how to find, information about, age ranges, information about the local area.
• All information is there. But would be interested to see what’s on the pages that aren’t in the prototype. Feels sections are good but would like to see how the content inside them is developed. Parents’ area would include official documents. Team page could be fun, include rollover images and graphics etc. Feels the site could be made to be quite creative.
• Having the top graphic on every page isn’t necessary. A bit excessive to have it on every page. Not a fan of the header typeface – picked up that ‘r’ looked a lot like ‘t’. The functionality works but the group section buttons take you too far down the page (page goes a bit under the header).
• The full-page width tiles are too difficult to read on a tablet – need to be made into tiles. Tiles would be a lot better – less scrolling, harder to miss information.
• Desktop navigation was easiest – can read the menu system a lot easier and get to where you want to go. However, if the icons were clearer, mobile prototype 1 would be good. Likes the menu at the bottom, frames nicely with the header fixed at the top. Hamburger menu is nice but prefer the icons at the bottom.
• Feels that page titles don’t generally lend themselves to get icons (especially the ‘day-to-day’ page). Agrees with George in that a calendar might work well as an icon for the ‘day-to-day’ page.
• Icons in general might be confusing, maybe use text instead.
• Logo should link to the home page.
• Brilliant start.

Ameer Al Ashhab

Ameer studies BSc Interaction Design and was one of the testers of my Stellardrive prototype.

Observation 

Ameer’s observation video is below:

The observation video gives the following insights to how Ameer used the prototype to complete the tasks set. Here are some key observations from the testing.

Testing – Desktop

• Tried to find session times initially by going onto the Day-To-Day page, then going onto the Home page and found them there. Read them all fine.
• Advanced slideshow by using the next and previous buttons. Did accidentally zoom screen whilst doing this.
• Navigated to the Groups page with ease and found the age of the children in the Jolly Giraffes class easily. Clicked on the class icon to go to the page (but first scrolled back to the top) and used the upwards arrow button to go back to the top of the page.
• Found the Day-To-Day page and read information about The Courtyard with no problem.
• Navigated back to the home page with no problem.

Testing – Mobile 1

• At first thought about clicking on the ‘Day-to-Day’ button to find the session times but remembered that they were on the home page. Read them with no problem.
• Used the next and previous buttons to advance the slides on the slideshow. When asked if he thought there was another way he said ‘no’ and thought it was a comfortable way to switch slides on mobile devices. Then he was asked to try and swipe the slides and he did.
• When asked to navigate to groups page, he clicked on the ‘Meet The Team’ button. Then clicked on the ‘Day-To-Day’ button, then clicked on the right page.
• When asked which page each button on the menu takes him, his answers were:
  • 1 – Home
  • 2 – Not sure (despite having just accidentally clicked on that button)
  • 3 – Groups, but he said that he is aware that there is ‘animal content on the page’ [referring to the name of the groups], but mentioned that initially he didn’t see the giraffe representing ‘groups’.
  • 4 – ‘Community’ – had to ask him ‘who might be at the nursery that the parents might want to know about?’ and he said ‘teachers’.
  • 5 – Parents
  • 6 – Contact
• Whilst on the groups page, found the age of the children in the Little Lions group with no problem.
• When asked to go to the Day-To-Day page he went onto the home page, tried to find the information I was asking for on there, then went onto the right page. On the page he found the information about the Nature Garden easily.
• When asked if it was easy to see the current page he said yes because of the hover colour.
• Went back to the home page with no problem.

Testing – Mobile 2

• Found and read the session times easily on the home page.
• Used the swipe gesture and the buttons to advance the slides on the slideshow.
• Had no trouble at all recognising the hamburger icon would open a menu and navigated to the groups page perfectly from that.
• Found and read the age of the children in the Mini Monkeys class. Clicked on the icon to navigate to the section and used the arrow to go back to the top of the page.
• Found the Day-To-Day page easily and found and read information about The Courtyard with no problem.
• When asked he said it was easy to see the current page.
• Navigated back to the home page easily.

Gaze Maps

The eye-tracking system was able to calibrate Ameer’s eyes and create a profile for him. Of all the gaze maps Ameer’s seemed to be the most accurate, but it only worked when the desktop and mobile prototype 1 was tested (for some reason it didn’t work when testing mobile prototype 2 – it’s possible that the reason for this is that he completed this test faster). It was good to see Ameer looking in the correct places for things like menus and content which perhaps suggests that the elements are laid out in a logical fashion and it was not difficult for him to find elements such as buttons and content. Note: I said it wasn’t difficult for him to find elements. not ‘pages’ – all that this proves is that elements were where he’d expect them to be (e.g. menu buttons at the top on the desktop prototype and the menu buttons at the bottom of the mobile prototype), not necessarily that he found finding the information he was asked to find easy.

Interview

Ameer’s interview is below:

Key points from the interview with Ameer:

• From a nursery website he’d expect to find out: what the nursery is about, what it does, contact info, Involvement of parents, tutors/staff, times of sessions/teaching, what kids do in sessions.
• Website has the all the info from what he’s seen so far.
• Nothing distracting or misleading.
• Info was easy to find.
• Giraffe icon was not logical, needs to be changed to group icon.
• Preferred hamburger overall. Was easier to understand, didn’t have to think about what to click on.
• If icons were more logical, would prefer bottom navigation. Only reason for not preferring fixed bottom navigation was the giraffe icon threw him off. Recommended small-sized text instead of icons to make it easier for users to navigate.
• Menu on the bottom easier to thumb on a mobile device.
• Desktop website easier to navigate overall. Would suggest fixed menu bar at the bottom if the giraffe icon was changed.

Robert Brown

Robert is my father and has worked in the IT service industry since 1984, moving to Norwich from Stratford-upon-Avon in 1984 aged 16 and working for MayDay (now known as MayDay Integrated Office Systems) from 1984 to 2010 when he left and formed BrownTech IT Services in November 2010, his own IT support company. He manages and supports IT and computer infrastructure for many East Anglian businesses, charities and several smaller schools and educational institutions including Nellie’s Nursery whom this prototype has been designed for. He completed the test at home a few days after the main batch of testing had been done. I felt that he might be able to provide some additional insight and also I felt that I needed a wider variety of people to test the prototype.

Observation

The observation video was recorded on my Nikon D500 which was mounted upside-down to my tripod above a table with the iPad on it. The benefit of using the D500 to record video is of course that is has superior video and audio quality compared to the webcam that the mobile testing rig has. The disadvantage is that of course eye-tracking cannot be done (not such a big deal in this test, more on this later) and my setup with the camera suspended above the tester on a tripod over a table wasn’t particularly comfortable for the tester as they had to sit on the floor to do the test. Some people might not think that this is actually a big concern but it is. Testers can get nervous or a little scared when asked to test a prototype where they will be asked to perform a number of tasks. They might fear ‘getting it wrong’ (technically impossible since if they ‘get it wrong’ that’s usually down to poor design which means that they don’t know what to do rather than them being inept) or ‘looking silly’ or ‘giving the wrong answers’ (which I assume means ‘giving the designer an answer that they don’t want to hear’ such as ‘I don’t understand why you did this’ or ‘I don’t like this’), so anything you can do to comfort testers before and during the test, ensuring comfort when completing the test, will help calm their nerves a little and might even give you better test results.

Going forwards, I’m not too sure how well this setup works. Granted, the video and audio quality rendered by the D500 and the Rode VideoMic is lovely, but it’s not terribly comfortable for the user to complete the test like this and the sheer size of the Nikon D500 especially with the microphone attached can sometimes unnerve people – especially when it is recording them. I could remove the battery grip and/or use a smaller microphone (or not use one at all) and a smaller lens to make the camera smaller to make it less obvious, but my parents and friends are used to me owning this camera and see me with it a lot, so they were fine with it. It might put others off a bit though. Recording in 4K resolution with the Nikon D500 absolutely eats its batteries as its CPU is working extremely hard to encode this video, it’s writing this video to the XQD card and the rear display must stay on whilst the video is recording (this is down to it being a D-SLR camera with mirrors – D-SLRs record video using a ‘rolling shutter’ which means that the live preview has to always stay on), so by having the grip attached with two batteries in the camera I can record for a lot longer than if I had just one battery and therefore record more testing, getting more data and potentially improving future work.

The observation video is below.

Key points from Rob’s observation video:

Testing – Desktop

• Firstly, went onto the Day-To-Day page, scrolled a lot on there, then tried looking on the groups page and found them there. Read them easily.
• Used the previous and back buttons at first to advance the slides.
• Found the groups page easily. Found the info for Jolly Giraffes. Clicked on picture to go to section. Clicked arrow to go back to top.
• Found Day-To-Day page and read information about The Courtyard with no problem.

Testing – Mobile 1

• Looked first on the Day-To-Day page, then tried tapping on the Meet The Team page, then went onto the Groups page and found the information there again.
• Used the buttons to move the slides, when asked if he thought there was another way he swiped.
• Tried tapping on the Meet The Team button first in order to get to the nursery groups page. Then tried tapping on the parents button. Then tapped on the right button.
• Was able to tell me the age of the children in the Little Lions class with no problems. Navigated by tapping on the icon and went to top of the page using the arrow.
• Navigated to Day-To-Page page fine and was able to read the information about the nature garden.
• Found the home page OK.
• When asked which page each button on the menu takes him, his answers were:
  • 1 – Home
  • 2 – ‘Can’t remember the title of that one’. Felt that the pencil icon doesn’t really have any kind of connection to ‘day-to-day’.
  • 3 – Groups (but didn’t know that at the start)
  • 4 – ‘I assume that was groups because it looks like a group of people’ Wasn’t sure what this was until he was asked which group of people the parents might be interested in reading about.
  • 5 – Parent information
  • 6 – Contact

Testing – Mobile 2

• Instinctively used the hamburger menu, found session times on the Groups page but only after having first tried to find them on the ‘Day-to-Day’ page again.
• Swiped the slides on the homepage to move them forwards.
• Navigated to the groups page very easily and found the information about the Mini Monkeys and navigated to the section. Pressed the arrow again.
• Found Day-To-Day page easily and told me information about The Courtyard.
• When asked to tell me the current page, he scrolled to the top of the page and read the page title. I hinted to look in the menu then he saw it was highlighted.

Interview

The interview was done in exactly the same fashion as the others were, the only difference is that I didn’t use a tripod to stabilise the camera (as it was already set up for mounting the camera upside-down facing the table and I didn’t want the hassle of setting up the tripod several times over), so the footage is a little shakier.

Key points from the interview with Rob:

• From a nursery website he’d expect to find out: what the children will do, structure, supervision (put parents at ease), contact details, times, the groups.
• Appears to have all the relevant content, nothing missing.
• Nothing distracting or misleading on any pages.
• Information easy to find, typeface fine. Icons not always obvious though.
• Prefer desktop.
• Hamburger menu better of the mobile ones. Told you the pages. Not always sure what the icons meant. Icons really need some text – that would be perfect.
• Bottom menu would be ergonomic to use on a smartphone.

Heather Brown

Heather Brown is my mother and does not work in the technology industry at all – she has worked in HR at Foster’s Solicitors since 2000 and before then worked as a PA in the NHS and doing similar secretarial jobs at Anglian Water before that. She has however always had an interest in childcare volunteering in Girl Guiding from about 1993 to 2013 (and later returned for a short period in 2017) and has also volunteered in several local special needs schools and nurseries. I thought that her input on what a nursery website should have would be very interesting to hear given her interest in childcare. I also wanted to see how somebody who says that they are not completely confident using technology finds the prototype. All of my other testers are people who use technology every day and are all confident with using it.

Observation

The video was recorded in the same way as Rob’s but unfortunately I set the exposure on the camera a little high and even with post-production colour editing in Premiere Pro, the content on the iPad screen can be a little difficult to read. Otherwise it’s done in exactly the same way.

Key points from the observation video:

Testing – Desktop

• Tried looking for the session times on the ‘Day-To-Day’ page first, then tried Parents Area, then looked in Groups and found them.
• Instinctively swiped the pictures in the slideshow (might be because I asked her to ‘move the pictures’ though)
• Found Groups page with ease, found the age of the children in the Jolly Giraffes class and navigated to the section by tapping on the picture. Didn’t notice the arrow, scrolled back to the top.
• Found the Day-To-Day page and read the information about the courtyard with no problem. Noticed error in the copy text.
• No problem going back to the home page.

Testing – Mobile 1

• Navigated immediately to the Groups page and found the information there.
• Remembered ‘Jolly Giraffes’ was a group, so naturally assumed that the button with a giraffe on it might be the groups page.
• When asked which page each button on the menu takes her, her answers were:
  • 1 – Home
  • 2 – Day-To-Day (but by process of elimination and only knew the page name as had been instructed to go there before).
  • 3 – Groups
  • 4 – Staff
  • 5 – Parent area
  • 6 – Contact
• Swiped the slideshow, found the buttons too after being asked if there was another way.
• Found Groups page, found the info for Little Lions. Tapped picture and tapped arrow.
• Found Day-To-Day fine and found and read Nature Garden information easily.
• No problem navigating home.

Testing – Mobile 2

• Instinctively used hamburger menu to find the Groups page and found the session times on there.
• When asked if she could think of another page that they might be on, she said she’d put them on the Day-To-Day page.
• Used swiping and the buttons to advance the slideshow.
• No problem navigating to the Groups page and found the age of the children in the Mini Monkeys class. Tapped image and tapped arrow.
• No problem navigating to Day-To-Day page or finding information about The Courtyard.
• No problem navigating back to the home page.
• When asked to tell me the current page, she scrolled and said she couldn’t see it. I hinted to look in the menu then she saw it was highlighted.

Interview

Again, the interview was done in exactly the same way as Rob’s. The video is below:

Key points from the interview:

• From a nursery website she’d expect to find out: opening times, age of children, who the staff are, what their qualifications are, contact details, what they might be doing, age groups
• Nothing missing, but would be good to have information about meals and food they provide. What sort, allergies. Also information about sleeping for young children (daytime naps) and also baby changing facilities.
• Nothing distracting or misleading.
• Information easy to read and find. Session times should be on ‘day-to-day’ rather than classes. Didn’t go down far enough on home page to see them – should be at the top of the home page.
• Desktop and mobile 1 easiest for navigation. Harder on mobile 2 – got to keep opening the menu. On others the buttons are available, can see where you want to go. Not a fan of the hamburger menu – icons are easier.
• Felt that mobile 1 would be ergonomic to use on a phone.
• Pencil icon doesn’t make sense – think about a ‘sun and a moon’ icon. The others were OK – not sure how to do groups page if the animal icon wasn’t used. Maybe words need to be used.

Emma McIlwaine

Bonus tester! Emma is the friend I have been working with to develop Inspix (and now another project too – details to follow!) but did not participate in this round of testing for the prototype. I met up with her at university on Friday 23rd and showed her my work. She said she preferred Mobile Prototype 2 because it was easier to navigate but noted that the hamburger menu animation should be a little faster and she suggested making the hamburger menu itself occupy about half of the device width to reduce the amount of blank space in the buttons on the menu. When using Mobile Prototype 1 she could only identify the Home and Contact icons and eventually identified the Parents’ Area icon, but none of the others were obvious to her. Like Naomi, she felt that she pencil icon for the Day-To-Day page meant ‘edit’ or ‘customise’ and like many of the other testers, she didn’t understand the use of a giraffe for the Groups page.

Conclusions from the testing

My eight testers were able to provide me with a lot of qualitative data to work with. Qualitative data is good because it is often detailed and testers can elaborate on specific points, but it is harder to analyse and put together. What I have done below is summarise what testers collectively liked and disliked about my prototypes and what some individual testers liked and disliked about them. From here, I am able to evaluate what needs to be done to address the faults that my testers found in the prototypes and how the next set of prototypes should look.

What my testers collectively liked about the prototypes

Desktop

• It was overall the preferred prototype.
• Testers felt that information was easy to find and there wasn’t anything distracting.
• Testers could read the information without a problem.
• Testers felt that generally the button labels would indicate that the nursery website would have all that needs to be on there (also applies to the mobile prototypes)
• Testers were able to navigate the menu system with ease with virtually no incorrect button presses.

Mobile 1

• Despite the icons not always being obvious on the menu, the fixed menu bar was the overall preferred navigation style on the mobile prototypes.
• Testers felt that this menu design would work well when using the site on a smartphone held in one hand and operated with one thumb.
• Testers could read the information easily and there wasn’t anything distracting on the pages because it used the same layout as the desktop site.
• Testers felt that the navigation system used in this prototype was unique and different – with some tweaks most felt that it would work better than the hamburger menu.
• The Home, Parents’ Area and Contact Us icons were all easy to understand.

Mobile 2

• Testers felt that this prototype was quick and easy to navigate around.
• Testers felt that they were able to quickly and easily find information because it uses the same layout as the other prototypes.
• Testers felt that the information was easy to read and that there wasn’t anything distracting on the pages.
• Testers generally enjoyed using this prototype but felt that the hamburger menu wasn’t really very special or unique.
• Testers all seemed to know exactly what the hamburger menu was (even if not by name) and knew that pressing the ‘three lines’ icon would open a menu. Even Heather who doesn’t often browse the internet on mobile devices knew that it was a collapsible menu.
• There were virtually no incorrect button presses when testers were testing this prototype.

What my testers collectively disliked about the prototypes

Desktop

• Many testers looked initially for the session times on the ‘Day-To-Day’ page despite the information being on the home page. Testers felt that the information was too far down the home page for them to notice it. This also applies to both mobile prototypes.

Mobile 1

• The biggest fault was that the icons weren’t always logical to the testers. Testers often struggled to find the ‘Day-To-Day’ and ‘Groups’ pages, often mistaking the ‘Groups’ page with the ‘Meet The Team’ page because they felt that the ‘Meet The Team’ page had an icon that was more appropriate for the ‘Groups’ page. The pencil icon for the ‘Day-To-Day’ page was not logical to most of the testers.
• Testers tended to remember certain anecdotes such as there being a group called ‘Jolly Giraffes’ so therefore the button with a giraffe icon might be the button for the ‘Groups’ page rather than actually navigating to the pages because they could figure out that the giraffe icon meant ‘Groups’ (for example) without knowing that little bit of context first.
• Testers also tended to press buttons on the menu based on pages that they had already visited – but this was frustrating for them as they first had to find out where each button took them before they could navigate the site effortlessly. This took time and was a poor experience.

Mobile 2

• Some testers commented on the performance of the hamburger menu button – on some browsers it appears that the user has to really press hard on the device screen or tap in a certain place to activate the menu, leading to slight frustration.
• Some testers just didn’t like the hidden navigation because they felt they had to keep tapping on a button to open the menu to move away from the page.

Individual positive thoughts about the prototypes

• Some testers felt that the prototype could easily be evolved into a legitimate nursery website.
• Some testers felt that the prototypes showed a promising start.
• Some testers liked the use of the Lorem Ipsum placeholder text.
• Some testers liked the ‘playfulness’ of the buttons on the fixed menu mobile prototype (mobile prototype 1).
• Some testers commented on how they liked the fact that I had designed an A-B-C test.

Individual negative thoughts about the prototypes

• Some testers didn’t like the font used for the headers – they felt it didn’t look professional and was too difficult to read (also, ‘t’ and ‘r’ are the same character).
• Some testers didn’t like the 100% container width tiles – they felt there were too big and rendered in the user having to do too much scrolling which might make them miss information. The tiles would look better arranged as a grid and there would be less scrolling. Key information could be put into these tiles rather than using them to house whole paragraphs of information.
• Some testers noticed that there were errors in the copy text (this was copied and pasted from the existing website).
• Some testers noticed that when you tap on class icon it will take you slightly too far down the page so that the title of the class group is hidden beneath the header.
• Some testers felt that the mobile navigation shouldn’t necessarily be either fixed or hidden, but a good mix of both.
• Although some testers instinctively swiped the slides on the slideshow to advance the slide, some mentioned that it was not obvious to them that the slides could be moved by swiping and that the absence of forwards and backwards buttons on the slides themselves was a surprise.
• Some testers suggested using labels or abbreviations (letters) on the fixed menu mobile prototype rather than icons as icons can be difficult to design and their ambiguity is subjective.
• Some testers commented that clicking or tapping on the logo should take the user back to the home page.

Turning qualitative data into quantitative data

The gaze maps give quantitative data but do not describe the qualitative data particularly well as they show different things. The qualitative data shown above is great but can be a lot to read and interpret, so what I have done is create some graphs using the responses I got from the interviews and what I observed my testers doing during the testing to better show where pain points where.

All of the data below proves the usability of the prototypes using quantitative data.

Note: the data below does not include Emma’s observations.

Testers who understood all of the icons on Mobile Prototype 1

Unfortunately, none of the testers understood all 6 of the icons without some hinting or background knowledge. 29% of testers found that only one icon was illogical and the rest could be understood and a massive 71% found at least two of them illogical. This clearly shows that overall the icons were not logical at all and would contribute to a poor user experience where the users would be struggling to understand what the icons represent, thus taking longer to navigate the website on a device that you tend to go to for doing things quickly on.

Specific icons in Mobile Prototype 1 which were not understood

No users in this sample of 7 had no problem at all understanding the Home, Parents’ Area and Contact page icons immediately (had Emma’s observation been included in this data then the Parents’ Area wouldn’t be in this list), but 31% of testers had trouble understanding the icons for the Nursery Groups and Meet The Team pages. Often the icon for the Meet The Team page would be confused for the Groups page and many people just didn’t understand the use of a giraffe as the icon for the Groups page, but some testers remembered the link to the ‘Jolly Giraffes’ class. 38% of testers didn’t understand the use of the pencil icon for the Day-To-Day page. The pencil was initially suggested to me as it could depict what happens ‘day-to-day’ in the nursery (drawing, activities, things like that), but it turns out that it was the icon that people understood the least. The icon was usually either not understood at all or mistaken for some kind of ‘edit’ or ‘customise’ button (which is often depicted by a pencil in other apps). This shows that half of the icons in Mobile Prototype 1 were not understood by over 30% of the testers, again showing that the icons were not logical and leading to the same poor user experiences described above.

Incorrect menu button presses (all three prototypes)

An ‘incorrect button press’ is a button press performed by a tester that did not take them to a page that they were instructed to go to – for example upon being told to visit the home page and then pressing on the button that goes to the Day-To-Day page would be registered as an incorrect button press. Note: tests where the user had to find content for themselves and looked for it on the incorrect page is NOT included in this data. These are usually all accidental and not deliberate. It shows how easy it is to navigate the website. Interestingly, there were absolutely no incorrect button presses on the desktop or Mobile Prototype 2 versions, but 13 on Mobile Prototype 1. The data shown in this graph is the result of the data shown in the pie charts above: illogical icon choices lead to lots of incorrect page visits which lead to frustrated users.

Page that users tried to initially find the session times on

The session times were available on two pages: Home and Nursery Groups. Interestingly though, open being presented with the prototype for the first time only 29% of testers instinctively thought to look for the session times on the home page, with 71% instinctively looking on the Day-To-Day page instead. Upon talking to a few of the testers afterwards, they said that they felt this was the most logical place for the information to go as the session times obviously directly link with the day-to-day activities of the children. Some testers said they didn’t see the information on the home page, so perhaps if I wanted them to find the session times there then it needs to be made more obvious. No testers thought to look for it on the Groups page which could mean that generally this is not a logical place for this information to go.

Instinctive method of advancing slides

When first presented with a slideshow on the desktop prototype, 57% of users used the buttons to advance the slides whereas only 43% thought to swipe. Despite the desktop website being tested on an iPad, perhaps the testers felt that they had to emulate their behaviour as if they were on a proper desktop or laptop computer with a keyboard and mouse and/or touchpad, so maybe that’s why they generally used the buttons. One tester said it wasn’t obvious to him that you could swipe the pictures to advance the slides and others tried looking for forwards and backwards buttons on the slides themselves. One tester said he didn’t consider the slideshow a ‘slideshow’, more of a ‘gallery’ because to him a slideshow is automated and a gallery is something that you browse through at your own pace. The 57% of testers who didn’t instinctively swipe on the desktop site had to be asked if there was ‘another way’ to advance the slides on the mobile prototype. Not all of them instinctively swiped. This shows that there is a potentially an issue here with it not being apparent to users that the slideshow can be swiped. It’s not a big issue because the next and previous buttons are obvious, but if it was obvious that the slideshows can be swiped then these buttons could be removed from mobile versions of the page, thus making it look cleaner and less-cluttered. The buttons are possibly too small to use on a mobile device and not completely ideal.

Preferred mobile navigation method

Interestingly, despite it being obvious that the icons in Prototype 1 were mostly illogical, my users preferred the fixed navigation style. I guess that this doesn’t necessarily mean that they liked the fixed navigation style that I had implemented, maybe it means that in a more general sense they prefer having fixed navigation on a mobile device. I can’t really declare that the Prototype 2 is the preferred mobile prototype when 57% of my testers disagree that hidden navigation is a good thing. I feel that testers felt with that different icons the fixed menu system in Prototype 1 could be a good navigational system. The fixed navigation means that you don’t need to keep tapping on an icon to open a menu to then navigate to a page and it can also be an ergonomic benefit, proven below.

Preferred the ergonomics of Mobile Prototype 1 to Mobile Prototype 2

86% said that they felt that to use on a smartphone, the ergonomics of Mobile Prototype 1 would be preferred. 14% represents one person whom suggested using a swipe menu instead of fixed navigation. He felt that this would be more ergonomic.

Preferred overall prototype

Overall, the desktop was the favourable prototype. Very few negative comments were made about it and those that were made tended to also apply to the other prototypes as they were to do with page elements like heading text and tiles rather than the navigation or general layout of the page. Users felt that the desktop site was easy and hassle-free to navigate. Asides from adding more content and making a few changes based on the feedback that I received, it seems that development of the desktop website in terms of adding features, changing the appearance and navigation does not really need to go much further and that most of my efforts going forwards needs to be directed more at building a mobile website that works for the vast majority of my testers.

 

Self-evaluation of my prototyping

After the first five tests had been completed on February 15th I recorded a short video explaining what I had found out as well as give my own opinions on what I feel should be on a nursery website and a little bit of background of why I made the design choices that I did.

I found that the testing that I have completed up to the point of writing this article has given me a lot of genuine feedback to work on to make the next prototype even better and steer the direction of the project.

Improvements since the Stellardrive prototype

I was keen to make sure that prototyping this website prototype produced more relevant and useful data than the Stellardrive prototype did. This prototyping session ironed out a lot of problems that the Stellardrive prototyping session had, notably:

• All testers did the same or a very similar test without any massive differences between each.
• I prompted testers much less than I did when testing Stellardrive.
• I tried to use a more diverse range of testers by introducing older people, one of whom does not feel she is massively confident with using technology (but is not a ‘technophobe’).
• The tests were blind, but once the user had used the desktop site they had a rough idea about what was potentially coming up for the mobile prototypes.
• All testers knew exactly what they were testing.
• The prototypes tested were evolutionary meaning that the next prototype(s) that will be tested will be based on the same codebase as these ones and will essentially be an ‘evolution’ of these prototypes. This means that I will be able to compare data and user pain points between prototypes and identify if progress is being made.
• The prototyping did provide the ‘concrete quantitative data’ that I wanted it to produce. Admittedly it didn’t do this through the eye-tracking and analysing gaze maps as initially anticipated, it did this by providing me with qualitative data that could easily be converted into quantitative data. For example, by observing user behaviour during the testing I saw that ‘a lot of testers first went to the Day-To-Day page to try and find the session times’. This was later converted into ‘29% of users went to the home page to look for session times and 71% of users went to the Day-To-Day page’ by analysing observation footage. I have done the same thing with the interview footage. I wasn’t able to convert any of the qualitative data from the Stellardrive prototype into quantitative data like this.
• The nature of the prototype that was being tested this time was a lot easier to prototype than Stellardrive because there were far fewer externalities to consider. The website could be tested and assumed that if the website were to be launched that is how a user might actually use it, but with Stellardrive the prototype didn’t take into account crashing the car, not concentrating, changing speed, changing gear and so on.

Thoughts about the collected data

As mentioned, I was able to get a good range of qualitative and quantitative data that really describes the whole picture when put together. Using all of the data I have collected, clear conclusions can be drawn and I am able to design the next prototype(s) taking my data into consideration.

No user persona

On February 12th when I wrote the prototyping methods and stated that I’d add a second dimension to the testing by having a user persona. In the end I didn’t use one for this round of testing. I felt that at this stage the aim was to find out if real users preferred mobile prototype 1 or 2 and what their thoughts were. I also wanted to find out if they could navigate the prototypes to find the information that I was asking them to find. User personas are all about trying to use the site as a member of the site’s target audience to find potential pain points, but at this point there is not enough actual content on the site to really try and use the site through the eyes of a persona. The use of the Lorem Ipsum placeholder text and the fact that several of the pages don’t exist in this prototype means that the scope of tasks for a ‘persona’ to complete is too limited at this stage. Future prototypes will be more complete and thus provide more scope for tasks. The persona doesn’t really need to be tested until a core of the website has been produced and then hopefully only minor pain points will be identified through the eyes of the persona which can be adjusted quickly towards the end of development.

No control

I also said that there would be a control again. The control was useful for the Stellardrive prototype where the user was expected to look at multiple places on the screen at any one time, or different parts at different times and for different lengths of times because it gave the reader of the test data and idea of what I would have expected as the developer of the prototype and the only person possessing the perfect knowledge to make the control. You have to imagine the Stellardrive prototype actually being a real product in order to understand why a control was helpful. I wanted to show that if you looked at the parts of the screen where I did for the same length of time that I did then the infotainment system could be classed as being safe and in real life this would mean that accidents on the road caused by usage of the infotainment system.

For a website where generally it doesn’t really matter where you are looking on the screen (it’s certainly not the equivalent of a life and death situation or even losing points in a game due to missing something), a control isn’t a necessity. The control would have looked a lot like some of the data that my testers produced, only done a little smoother and faster without hesitation as I know my way around my own prototype. In this example, it was important to know how I expected people to use the website, but it didn’t really need to demonstrated in a control test. The benefit of using a control though would have been that I could compare how my testers used the site compared to me – its developer – but it’s not as important as it was in the Stellardrive prototype. It might be something I consider for a future prototype of this project though.

Thoughts on eye-tracking

As mentioned in the write-up, I was informed by the UX lab staff that the eye-tracking doesn’t work as well on the mobile device testing rig as it does on the desktop computer due to the location of the hardware eye tracker not being inline with your eyes all the time. Visiting software company Redgate on February 20th (a few days after the prototyping for this was completed) we discussed the purpose of eye-tracking in UX testing and they said they were surprised that people were so keen to use it. We talked about the data I had collected and how I had found it hard to really analyse it and get anything meaningful out of it and they said that if you test websites like my prototypes with eye-tracking software and hardware these are the types of results you can expect. When I told them how helpful the eye-tracking had been in my Stellardrive prototype and how I was slightly disappointed with it for this prototyping session they said that the reason why it was more helpful in the Stellardrive prototype was because I was using it on a system where the user is expected to only look at certain elements for short periods of time else the consequence could be fatal. They themselves said that the only use they could think for eye-tracking in UX testing was if you were trying to test something like a flight control system. They said on websites like my prototype all it does it put big dots on the screen and doesn’t really produce any meaningful data unless you are wanting to find out if the tester is looking at a specific part of the page.

I think that’s it really, my prototype doesn’t require users to look at ‘certain parts of the page’ like the Stellardrive prototype did. I have to say that in this example I wasn’t really able to critically analyse the gaze maps like I could when I analysed the Stellardrive test data because the data wasn’t really showing me a lot. I found watching the observation videos provided more data because I could see exactly where users were tapping and during the testing some of them were telling me what they were doing so the audio was vital too – which is why it’s a shame that the audio wasn’t recording for the first two of Celestin’s tests.

Doing some research on the application of eye-tracking in UX, I found some interesting points suggesting that eye-tracking is not useful. Take a look at the extract below from an article on UIE.com:

Not every participant can work with an eyetracker. Depending on the hardware, people with a variety of attributes automatically are disqualified from eyetracking. Everything from contact lenses to long eye lashes can get in the way of the device working properly.

Third, they reduce the amount of time you actually collect data from your users. Getting a participant set up and calibrated with the device can take time away from learning about your design. The most valuable piece of any usability test is the time the participant is interacting with your design, not setting up the measurement equipment. What’s worse is many devices lose calibration quickly, forcing the test to stop and the participant to spend more time futzing with recalibrating. This tool time is distracting and not adding to the session’s value.

Fourth, the results are really hard to analyze. The colorful heatmaps are cool (or warm?) to look at, but what are they actually telling you? When someone is gazing at something, is it because they want to look there? Or because the page made them look there? Or because they are resting their eyes there?

I experienced all of these during my testing session. I didn’t want to waste time trying to calibrate all of my testers’ eyes because I was working with limited time. If a profile couldn’t be made for them, I just used another profile and hoped for the best. This means that my eye-tracking data was not as accurate as it could have been as my testers’ eyes were often not calibrated correctly. The more time a tester spends interacting with the prototype, the more observational data you can get and the more you can learn from that. You can visually see what they are tapping on if you place a camera above them during the test and record it all. You can hear what they say to you and when they’re ‘umming and arring’ over various things. The extract makes an interesting point about the validity of the heatmaps (or gaze maps, in my case). Sometimes it is not always easy to understand exactly what is being looked at when performing eye-tracking on something like my prototype.

The article then goes onto the raise the following interest points:

When we first started conducting eye tracking, we noticed some interesting behaviors:

Participants often would acquire the scroll bar without looking at it. They’d move their mouse over to the right edge of the screen and start scrolling, but their gaze wouldn’t leave the center of display. It seemed they were using their peripheral vision to acquire and use the scroll bar.
Participants would orally tell us they couldn’t see something their gaze was focused on. (Women in my life have referred to this as “Male Refrigerator Blindness” — the inability to see something right in front of you.)
Participants often would click on objects they barely gazed at. They’d focus their vision on some part of the screen, then move their mouse to some place else to actually click.
From this, we began to question what the eyetracker was actually trying to tell us. It seemed to us that what the user focused their gaze on was not necessarily what they were seeing. So, if the eyetracker doesn’t tell us what a user sees, what does it tell us? I’m not sure.

It’s possible that I experience this in my own testing. Perhaps some users were clicking on things that they weren’t really looking at and so the eye-tracker missed this. It could have been down to the poor calibration, but quite often in my gaze maps I could see that testers were tapping on the menu buttons but the gaze map wasn’t picking it up at all. It seems that it doesn’t always pick up things that your testers are just quickly glancing at.

I certainly didn’t find eye-tracking as helpful this time round as I did when testing Stellardrive.

Going forwards I will carefully consider whether eye-tracking is really required. It did provide me with some data and combined with the observation videos and interviews it provided an extra layer of data for some of the users that it worked correctly on, for example I was able to consider whether large spots for slightly prolonged periods of time could suggest hesitation and then judge whether this was because of nerves of the tester or that they were confused and had to think. This kind of insight into my own prototyping methods (if the large spots are to do with the user’s nerves) cannot necessarily be provided by interviews or any kind of qualitative data or percentages alone. The chances are that for phase 2 prototyping when the next prototype is tested I will use eye-tracking on the mobile device testing rig, despite it not always being useful for this type of testing and my Nikon D500 setup providing superior video and audio quality (thus making analysing observations easier) but I will likely not attempt to analyse it as much unless I find anything particularly noteworthy.

Moving forwards: ‘Mobile Prototypes 3 and 4’ and Phase 2 Testing

Mobile Prototype 3

The next prototype will be built upon the extensive feedback that has come as a result of this testing. At the time of writing one wireframe of a potential ‘Mobile Prototype 3’ has been sketched which could be developed. It is essentially a blend of mobile prototypes 1 and 2, featuring a Menu button fixed at the bottom of the page that the user can tap at any time to bring up a menu that fills about half the screen and has buttons to the other pages that feature both labels and icons that the user can tap on. This should eliminate most of the negative feedback generated from testing mobile prototype 1 whilst maintaining the simplicity of 2.

When I met Emma on Friday 23rd after she had given me some feedback on Mobile Prototypes 1 and 2, I described her how I envisaged Prototype 3 looking and how it would help remedy some of the problems found in this round of testing whilst retaining icons and the favoured ergonomics of Mobile Prototype 1. I told her how one of my testers (Celestin) had described a hamburger menu that was activated when the page was swiped left or right, but she mentioned that most modern mobile browsers use these gestures already to move back and forth between pages (this is certainly true for Safari on iOS and Edge on Windows 10 Mobile). She suggested that I made another prototype (‘Mobile Prototype 4’) that featured some of the most commonly-required pages appearing as icons at the bottom of the screen and some less commonly-used pages appearing in a hamburger menu which would also be activated from the bottom of the page to maintain the ergonomics. We looked at the survey results that I collected a few weeks ago and noted that the most common reasons for visiting a nursery website were to find out information about them (available on the home page) and to contact the nursery (available on the contact page). Given that these two pages had icons in Mobile Prototype 1 that everybody could recognise, the buttons to these pages would be icons that remained at the bottom of the page and the other pages would be in the hamburger menu. Emma used the example of the current Facebook app for iOS as an example of where this design works well. I am not yet certain that I am going to develop this prototype, but I certainly feel that it was a very good suggestion that will remedy many of the issues discovered in prototype testing in Mobile Prototypes 1 and 2.

Phase 2 Testing

Phase 2 Testing will be the testing of the next prototype(s) and will hopefully take place in March when the next prototype(s) have been developed. Eye-tracking and the mobile device testing rig may be used, or I may use my Nikon D500 setup instead to record observations. It’s too early to tell at this point in time.

If there is no Mobile Prototype 4 then the testing will likely follow a similar A-B-C method, comparing the desktop website against mobile prototype 2 and the new mobile prototype 3 to determine which the user prefers. Mobile Prototype 3 will almost certainly be a replacement for Mobile Prototype 1 and if Mobile Prototype 3 is preferred to 2 then 2 will likely be discontinued after Phase 2 Testing is completed and development of 3 will be continued as the mobile version of the Nellie’s Nursery website after user testing dictates that this is the preferred mobile design. However, as I don’t envisage much development of the desktop website to have happened between now and Phase 2 Testing, I may use a simple A-B test between Mobile Prototypes 2 and 3.

If a fourth mobile prototype is developed, then the testing will likely also follow an A-B-C method but this time comparing the desktop website against Mobile Prototype 3 and 4 with the older 1 and 2 prototypes forgotten about. Mobile Prototype 4 would be an evolution of Mobile Prototype 2. The testing would determine which mobile prototype the users preferred and then whichever came out more favourable would be continued to be developed as the mobile version of Nellie’s Nursery website.

Talking to UX designers at Redgate when I visited on the 20th about how many should test your prototypes, they recommended that you keep on testing and iterating until bugs and pain points are no longer noticed and they keep getting more and more people to test your prototypes until you are no longer surprised to hear what they are saying. By the time four or five people had tested my prototype I wasn’t surprised by what they were saying or doing and as mentioned before, five testers is often considered the magic number. However, Redgate UX designers disputed this and said that is not a solid number set in stone and you should also keep changing your testers every time you test a new iteration so that fresh eyes and a wider group of people are testing your work. For this reason, I am very keen to find at least five new people to participate in Phase 2 Testing (hopefully from lots of different courses at NUA) and have already begun advertising for this in the NUA Networking Society Facebook page. An hour or so after I put some information on there about it, I’ve had three people from three separate courses express an interest and I know some of my friends who do graphics are also keen, so I am hopeful that this will be achievable.

Bibliography

UX Planet. (2017). UXer’s quick guide to eye tracking – UX Planet. [online] Available at: https://uxplanet.org/uxers-quick-guide-to-eye-tracking-edf70bffd03d [Accessed 23 Feb. 2018].

Uie.com. (2006). Eyetracking: Worth The Expense? » UIE Brain Sparks. [online] Available at: https://www.uie.com/brainsparks/2006/06/13/eyetracking-worth-the-expense/ [Accessed 23 Feb. 2018].