Eric's Chi-Leaders

Further Evaluation (prep)

with 11 comments

The next part of our project will consist of creating a digital mock-up version of our interface.
With this mock-up we can conduct another round of usability testing.
The difference with last time is that we will be looking for more tangible data now.
We are especially interested in numerical representation of how satisfied the user is, how much time it took him/her to learn to work with our application, how many error messages they encountered, ….

Evaluation Goals
The most important goal of our new round of user-tests will be to find out if our previous adjustments have indeed made the interface and system more understandable overall.
One very important indicator of this is the amount of clicks users would need to make in order to accomplish their tasks.
If a user needs an exceptional large amount of clicks to perform a task, we can almost always assume that he/she has been looking around in the wrong places first, before actually finding out what to do.
Another red flag would be users staring at the screen for a prolonged period of ti;e, not really knowing what to do.
As with the previous indicator, we will log any such occurrence for further evaluation.

Our primary audience is “anyone using social media“. But since this is a very broad area to cover, we have decided to narrow it down to the 14-26 age range.
Finding people to test our application should not be too difficult. We can pretty much use everyone on campus, people in the streets and/or family members.

In order to conduct our tests uniformly, we have prepared a general workflow which we are going to use in all our experiments:

Evaluation guidelines:

1. Outline

The goal of this step is to inform the user about the goal of our application without steering them in a certain direction. We should try to be factual, and show no emotional investment in the application itself.
As our professor pointed out, some people might be kinder because they don’t want to hurt your feelings.
Friendly as that may be, we are looking to improve our applications and not our egos.
Perhaps test-subjects will be more candid if we only tell them we need to evaluate the application, and hide the fact that  we also made it.
Overall the initial spiel would sound something like this:
We have to evaluate a user interface for our CHI-course. It would be really cool if you could test the interface and then answer some questions for us. This applications goal is to allow users to read news-items and draw their remarks directly on top of the articles, much like graffiti.

2. First test phase

The user is asked to work their way down a list of routine actions.
We have to make sure that the assignments are described in a way that most people can understand them.
One way to achieving this could be to provide a little situational sketch to go along with each task.
For example: “You overheard your friends talking about a funny drawing they made on top of the Tsunami article. You decide to check it out.

Some thing we would like the users to try are:

  • Open an article
  • Write on top of an article
  • Modify topic preferences
  • Upload an article and invite friends to join
  • Search for articles regarding the Tsunami in japan
  • Add an article and make it public. Then remove one of the people that are drawing
  • Change your email settings

While the user is doing this, we will keep track of the time, the number (and types) of errors encountered and whether or not the user is going in loops.

3. Second test phase

At this point, the user has just finished his/her assignments. The interface and behavior of
the system should still be fresh in mind. This is a good time to ask them to fill in a short
questionnaire about their experience with our software. We have chosen for the Questionnaire
for User Interface Satisfaction – Chin et al, Maryland 1988.
We want to be able to correlate the type of user with their appreciation of the application or
identify problems specific for a certain demographic. Therefore, we will add some questions
asking the user’s age, amount of time spend in front of computers, the user’s educational
level, knowledge about social media, … .

User test Questionnaire

Live long and prosper,


edit: Blatantly forgot to write out the part about our audience and what we want to learn. Thanks to Kristof for pointing this out.


Written by chileaders

30/03/2011 at 12:51

Posted in Meeting, Men-At-Work

11 Responses

Subscribe to comments with RSS.

  1. Is there a reason for choosing the QUIS?
    Will you find new people to test the application or will you reuse the people you’ve used in the paper prototype tests? If you reuse them, you probably can’t say that you have to evaluate ‘a’ user interface, because they’ll know it’s yours.
    It might not be a bad idea to look for complete strangers, with no relation to you whatsoever. They will be less ‘friendly’ to you and might give a more objective result.

    The Chili Peppers

    31/03/2011 at 15:03

    • We chose the QUIS because it seemed to fit better with our project than any of the other questionnaires.
      This might be a bit of a vague explanation but most of these other lists contain questions on how well the tested system integrates with their professional life or how it helps them to be more productive.

      Since we are making an application that is more of a game, we do not really see the use of these types of questions.


      01/04/2011 at 15:13

  2. I like the thinking in your outline, especially the part you suggest hiding the fact you also made the application.
    I think there is one thing missing in your post though. Who is your test audience, and how are you going to find them?

  3. Agree that it is a nice touch that you will hide that you developed the application.

    Overall, this description is very verbose, but does not provide much rationale. For instance, your application is game-like: then why focus on number of clicks or time? I’m not sure that is a very relevant measure for your application?

    What exactly will you ask the users? If it is nothing but “You overheard your friends talking about a funny drawing they made on top of the Tsunami article. You decide to check it out.”, then they probably will not try the 7 things you list in the bullets?


    01/04/2011 at 18:32

    • We consider our application to be game-like in the sense that it is not intended to aid in a business environment. Also the drawing itself can be seen as a “game”.
      On the other hand, the navigation and the act of actually opening an article should be fast and uncomplicated.
      We believe that measuring the amount of clicks user need in order to perform such task would be a good indicator on the transparency of our navigation system.

      We are planning on providing the users with a small scenario describing the task at hand. This will be done for each of the items in the bullet list.


      05/04/2011 at 00:55

  4. I also like the idea of not telling the users you made the application.
    I do wonder why “Change your email settings” is one of the testing scenarios. It doesn’t seem important for the general use of the application. I would keep such details for later. The other scenarios seem like a good combination to represent the general usage of your application.

    Tanguy Monheim

    02/04/2011 at 09:52

  5. The description of your test phases is quite complete. Some small details could be added however. For example like Duval suggested: will you ask for general scenarios or more specific ones? Also what will you time? The time to execute all suggested scenarios or start timing for each specific one?

    Not really important to include into a blog post like this, but think about what you mean by “Errors”. I suppose an error is when the user goes to a page that doesn’t bring him closer to his goal? But what about a user asking: “What should I do now?” – Is that an error as well?

    The user test questionnaire is looking good. The extra information you are asking is very useful. Slight side remark: in the “Do you know/use” it’s probably too vague for the users to leave empty space. What are they suppose to write, just “Yes, I know it, but I don’t use it.” Or a lot more elaborate? If they are supposed to just answer “Yes/No”, I would just make a table (with a column for “know” and “use”) in which they can mark what applies to them.

    Bart Bottu

    02/04/2011 at 12:55

  6. I’m not very sure why you choose to narrow down the test-group from age 14-26. I think you will miss a lot of valuable information when testing the application with people a little bit older. They aren’t propably as skilled as a 18 year old student so you can really test the ‘simplicity’ of your application on these people.
    The user test questionnaire looks good and I think it will provide you a lot of usefull information.


    02/04/2011 at 15:18

  7. Most things are mentioned in the comments here, but I also would like to notice that the idea of not telling the users you made the application is a very good idea. I have one small question about your questionnaire though. Why do you want to know the level of education of your users? I think the age and hours spent using a computer are more important and might influence your results harder.

    Team SachiniNews


    03/04/2011 at 23:09

  8. I also like the idea of not telling the test users that you made the application.
    I do wonder though why you narrowed it down to 14-26 year old users. What was the reason behind choosing this range?

    Dieter van Loon

    04/04/2011 at 14:37

  9. It indeed seems like a good idea to hide the fact that you made the app from the test subjects.
    @The CHIli Peppers: It certainly is possible that complete strangers might be less inclined to give nicer feedback than acquaintances, but it might also go the other way. A complete stranger might be less comfortable to give honest negative feedback than a acquaintance.
    I like the demographic additions to the questionnaire. If the test subjects have to fill it out themselves it is perhaps still a little vague. But simply adding a dotted line or check boxes would already help. And for the first six questions of QUIS you wight want to add some text instead of only answers. But that’s all superficial.
    The question “Do you usually read a news papers?” does it apply only to paper news papers, online or both? Further specifying this question might be useful.



    04/04/2011 at 21:57

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: