Eric's Chi-Leaders

Test subjects analysis

with 5 comments

Subject 1 (member from own group)

Using a basic menu below with 3 buttons: articles, inbox and options.
For us this seemed to work perfectly (ofcourse) but members from other groups have shown us the flaws.

Subject 2

Still using the menu with articles, inbox and options.
Tools to adjust an article are always visible.
The main article menu shows several windows for each topic plus a list of topic-related articles per window.
An add article button is placed next to the article search bar in the article menu.

User chooses an article and doesn’t know whether he is able at this point to adjust the article, user doesn’t adjust the article.
Clicks on a friend and realizes this shows him basic info about that person in facebook style.
Clicks on the article button in the menu below and it presented with the articles list again.
Clicks on the door handle and doesn’t expect this to do anything special, which it doesn’t (apart from a small joke).
Selects ‘add article’ next to the article searchbar, uses the functionality there to make a new article.
Uses ‘add friend’ on the left to add one of his (facebook) friends to the article.
Uses the administrator article options to make adjustments to his article.
Goes to the options in the below menu and realizes this is a list of several account options.
Selects the inbox button and thinks this is a list of private messages sent by other users.
Clicks on one of the messages, which presents him an article (possibly adjusted by one of his friends), uses the pencil to adjust the article.
Closes the article and is now presented with an empty screen, with the tools and the below menu still showing.

Subject 3

No adjustments made yet.

Asks us how the application is started: using facebook or a website.
Clicks on the friends topic in the article menu and gets to see an article written by one of his friends.
Thinks that the cross button next to every friend on the left side of the screen is used to remove certain overlays from the article. This actually removes users from an article.
Uses the tools to adjust the article.
Closes the article and is now presented with an empty screen, with the tools and the below menu still showing.
Our test subject suggests he would rather have gotten the article menu instead of an empty screen, we agree this is a good idea.
Goes to his inbox and is presented with a list of personal messages, each accompanied by an “eye” symbol or a “pencil” symbol.
Realises the meaning of the symbols is read or write but doesn’t know if this means you can only read the article or if one of his friends read an article, same for the write symbol.
Goes to his account settings and is unsure what ‘topic preferences’ means, doesn’t make the connection to the topics in the article menu.
Asks if he used all of the app’s functionality and whe tell him he hasn’t added an article yet.
Quickly finds the add article button and states the options to add an article are clear.
Uses the main close button on the top of the screen to close the application.

Subject 4

Since our first 2 subjects waited a long time with using the tools to adjust an article (and stated this wasn’t immediately clear) we have added some colour to our tools illustration, making them stand out more, plus an extra tooltip.
When starting the application or closing an article the user now immediately goes to the article menu instead of an empty screen.

Subject is already looking at the articles menu but clicks on the articles button anyways, nothing happens.
Clicks on a topic name and receives a list of all articles related to that topic.
Selects one of the articles, since this article had an eye symbol in the list the user can only read the article and not adjust it. Although this article is read-only the tools are still visible (but disabled). Since this is confusing we make the conclusion that the tools should be invisible while reading a read-only article.
User clicks on the door handle and expected this to be an application easter egg.
Is the first person to use the search article functionality.
Goes back to the article menu.
Fills in an article name in the searchbar and clicks on the add article button. The searchbar has actually no connection to the add article functionality.
In the add article menu the user clicks on the delete article button. This can’t happen since the article hasn’t been made yet, we make sure this is invisible in such a situation.
Goes to his account settings.
Wonders what ‘notify by mail’ means.
Goes to his inbox and clicks on an article with a pencil symbol, wonders whether he’ll be able to adjust this artcile.
Uses the pencil and the eraser (first to use this) to adjust the article while ignoring the two other tools.
Closes the article and goes back to the article menu.

Subject 5

The tools to adjust an article are always invisible now, unless the user is reading an article he/she can adjust.

Subject clicks on a read-only article and is unsure what the list of users on the left side means.
Closes the article and selects the friends topic in the article menu.
User clicks on another read-only (“eye”) article and wonders why he can’t adjust the article.
We explain the use of the eye and pencil symbols and offer him to use an adjustable article from now on.
Uses the pencil and eraser to adjust the article, closes the article and goes to his inbox.
Assumes the inbox is a list of articles made by his friends.
Wonders how one can be the administrator of an article.
Wonders how one can add an article, looks for this functionality, finds it in the article menu and assumes he can now put an article source in the searchbar to add it.
Adds an article without taking a look at the add article menu.
Clicks on the administrator options button next to his article name.
States the use of our application was initially unclear but after he experimented with it, it was.
Subject 6

Some adjustments were made. The biggest adjustement is the menu below. It now has the following 3 buttons: articles, add article and (account) options.
The add article button used to have a bad positioning in our app since most people thought they were able to fill in their article source or article name in the article searchbar. Most people also had to look too long to find this functionality. Therefor we have moved it to the 3 button menu making it always visible to the user and in the meanwhile avoiding the confusing related to the searchbar. A smaller adjustment is making the magnifying glass available at all times.
Extra tooltips have been added to the add article menu to give more information about the options there.

Clicks on an article in the article menu.
Clicks on the door several time, then realizes it’s just an easter egg.
Is the first person to use the ‘spray grafitti can’ to adjust the article.
Goes to his account options and understands the different options there.
Goes back to the article menu.
Selects the ‘friends’ topic in this menu.
Goes back to the articles menu and clicks on an article in a different topic.
Uses the magnifying glass to zoom in on the article’s text.
Clicks on several users in the users list.
Selects the add article button and uses all add article options to add an article, likes the tooltips with extra information.
Selects another article in the article menu.

Subject 7

No further adjustments made.

Selects the friends topic.
Selects a read-only article in the presented list and while reading the article wonders why the tools are unavailable.
Closes the article and uses all add article functionality to add one.
Uses the tools to adjust the article and states it os clear that there are 4 tools to use in total.
Is the first to use the scrollbars, we assume previous users knew this was clear.
Selects the door handle and likes the small joke. Uses the search article functionality without any problems. Uses the application close button.

States that the article menu is a bit too disorganized and prefers a simple list of articles instead.

Subject 8

Extra tooltips are available that are presented each time a new article is opened and state whether the article can be adjusted or not. This way we hope that it becomes clear to the users what the difference is between the pencil and eye symbols.

Subject 8 uses all functionality and states everything was clear from the beginning.

Special Subject 9

Assumes the friends topic contains a list with friend names.
Selects a friend (actually an article but since we didn’t write out full article names this caused the confusion).
Reads the article, makes some adjustments and clicks the add article button to save his changes.
Is presented with the add article menu and now realizes this is used for something different.
Uses add friend to add friends to the article and assumes his friends receive a notification.
Closes the article and goes to his account options.
Goes back to the article menu and selects the “local” topic.
Selects a read-only article and uses the magnifying glass on it.
Thinks it is unclear in the article menu when an article is adjustable or not, we agree and state that the next version will have a pencil or eye symbol next to each article in this menu (and not only in article lists).
Uses the article searchbar, assumes that in this list a pencil symbol means that an article can be adjusted.

States that one big list in the article menu instead of the different topics is better. We agree and adjust the article menu. Initially it is best to keep our app simple, if there is a future need to have a distinction between topics we can always add it then.

3 non-computerpeople
When testing the system on family members a very interesting conclusion was made (independently) by all 3. They all assumed the difference between an eye symbol and a pencil symbol means that they have read that article or that they have adjusted that article. This is a very different result than we got while testing with the computer scientists!
Most of them didn’t understand what adding a tag to an article was plus the public versus private options were also very confusing.
They stated that they felt limited by a system that also has read-only articles: they just wanted to add their comments to any article!

For the next iteration we conclude that it is best to make every article adjustable and public, thus removing the public vs private and read-only vs write/read distinction. This is to initially make all users able to see and adjust every article, instead of limiting their options. Again, we can always add these limitations later if there is a need to do so. Focusing on the main idea and attracting users for now is more important.

note: we will add a youtube link to our chileaders account soon with several videos of the testing in action 🙂

Written by chileaders

25/03/2011 at 00:45

Posted in Uncategorized

5 Responses

Subscribe to comments with RSS.

  1. I see you made adjustments like every two test persons. But then how can you be sure the comments from only two users were really a big problem. And are you sure the solution for them was good?
    About the tooltips, they’re a great way to explain what something in your application does. But since you need tooltips to do that, isn’t there another deeper problem?

    • The adjustments made on short term were basically ‘common sense’ to the test subjects and us. For example not showing the tools when no article is presented, giving the tools a bit more colour(instead of just black on white) or presenting the user with the main articles menu instead of an empty screen. So we made those pretty early on.(usually every 2 or 3 test subjects)
      The long term adjustments were made after sufficient feedback and consensus in the group, like changing the 3 button menu which we did after 5 tests and a thorough discussion in the group.

      Agreed on the fact that the tooltips were used as a temporary solution to a bigger problem, if you read our conclusion you will see that the distinction between public/private or read/write will not be made anymore, thus also removing all of the tooltips involved.


      25/03/2011 at 12:12

    • During tests with our first paper prototype, we had a button with a label that was rather confusing, which was pointed out by almost all of our first test subjects.
      Instead of changing it after say, four tests (we did eight tests in total), we argued the we couldn’t change our test setup before we had gathered a significant amount of feedback.

      If I remember correctly, we finally changed it after the fifth or sixth test or so, although almost all of the previous ones had indicated the flaw. In retrospect, If we would have changed it earlier, we could have already tested the actual solution on a greater number of subjects (which would have allowed for a second change if the problem persisted).

      So although it might give rise to ‘false positive’ flaws, I believe that rapid iterations give you the possibility of trying out more solutions and might in the end solve problems more quickly.


      25/03/2011 at 23:48

  2. The difference between the CS and non-CS people is interesting. This shows clearly that it is necessary to test the application on other people too. Luckily you did those extra tests and can now further improve your application.

    Tanguy Monheim

    26/03/2011 at 10:45

  3. I also don’t see a real problem in changing small elements of the paper prototype during the evaluation, as long as the changes aren’t too big. By making these changes you can shift the focus from the test subjects away from these small problems, to the more important problems and thus save some precious time.
    The differences between CS and non-CS people are indeed interesting. It points out to us that although things may seem obvious to us, they actually aren’t obvious for the ‘general public’ or that they have a total different view on them.


    28/03/2011 at 22:34

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: