Hi!
I've got some equipment ordered so we can do some proper video-based usability tests of Fedora Community. In preparation, I've put together an initial stab at a task list for the usability tests. Since I think we want to capture the breadth of the site, but we also don't want to scare our testers away from ever helping us out again by taking up too much of their time, I put together two sets of tasks to assign to testers - 5 tasks per set. They shouldn't take any more than 20 minutes to run through. I think if we could get at least 5 testers per task set we'd be in good shape.
I also tacked on some introductory questions so we could get a feel for the tester's previous Fedora Community experience or their initial impressions of the site. There's also a set of closing discussion questions to get the tester's feel for the site after running through the tasks.
What do you think? Am I missing any essential functionality coverage with these tasks? Is the balance between the two task sets fair? (I tried to start both sets off with easier/quicker tasks and balance the number of more involved tasks per each.)
~m
Test Requirements: • Users must have an FAS account. Or we need to set up a test account so they can view the people search portions of the site and so the package maintenance sections of the site are meaningful. (If they own 0 packages we'll not be able to test some of the core functionality I think we want to.) • Package portions of the test are going to be way more useful if we have package maintainers as test users.
Introduction / First Impressions (All questions on front page) • Have you seen this site before? ∘ (If yes) Where did you hear about it? • Have you used this site before? If so, what have you used it for? ∘ (If yes) How did that go? ∘ (If yes) How often have you used this site? • What are your first impressions, looking at this site now? • What do you think this site is? • What would you click on first? • Any other comments you'd like to make about this site? • What do you think of this page? Where would you click first?
TASK SET I
People
1) View your profile on this site. What do you think about it? ∘ Is the information accurate? Do your group memberships seem accurate? Do your package affiliations seem accurate? 2) You've just spoken with Dennis Gilmore and he's asked you to open a ticket for him in a fedorahosted.org trac instance. In order to cc Dennis on the ticket, you'll need to figure out his FAS account name. Where would you go to look it up?
Packages 3) Do you have any unpushed updates? Where would you go to look this up? 4) How are your builds doing? Where would you go to look this up? 5) Let's say you have a friend who is a little bit behind the times - she is running Fedora 9. There's an annoying Inkscape bug that bothers her - after searching the internet for a bit you found out the bug is fixed in Inkscape version 0.47. ∘ Is Inkscape version 0.47 available in Fedora 9 for your friend? ∘ If Inkscape version 0.47 is not available for Fedora 9, can you download an SRPM here to build it for her?
TASK SET II
People 1) You need help with yelp. Who's the package maintainer? 2) Luke Macken is going away on vacation for a couple of weeks and he's asked you to look over his packages while he's gone. How many packages does he have, and do any of them need attention? Where would you go to look this up?
Packages 3) How would you file a bug against NetworkManager using this system? 4) What has changed in the 'ruby' package over the last few updates? Where would you go to look this up? 5) How's the karma of your current testing updates doing? How would you check up on this in Fedora Community?
Closing • How well does Fedora Community support your package maintenance workflow? • Would you use Fedora Community to help maintain your packages? Which parts? • Is there any essential functionality missing from Fedora Community that would help you maintain your packages more easily? • Did you learn anything about Fedora Community's functionality through this test? • What do you think is the most useful function of Fedora Community? • What are the least useful functions of Fedora Community?
Rejected Tasks (I came up with these but decided they were less important than the ones I selected)
• How many bugs are open against python right now? • You just found out from an artist friend that there's a handful of supplemental plugin packages for the Gimp that let you do cool things. Look them up - what are they? • You see someone in freenode IRC with the nickname 'halfline' and you're not sure what his/her real name is. Can you try looking it up here? • You'd like to meet with Kushal Das to discuss one of his packages, but he's in a different time zone so you're not sure when would be a reasonable time. Can you figure out your time difference with him and work out a good time to propose a meeting with him on IRC?
On Wed, Aug 19, 2009 at 03:10:25PM -0400, Máirín Duffy wrote:
Hi!
I've got some equipment ordered so we can do some proper video-based usability tests of Fedora Community. In preparation, I've put together an initial stab at a task list for the usability tests. Since I think we want to capture the breadth of the site, but we also don't want to scare our testers away from ever helping us out again by taking up too much of their time, I put together two sets of tasks to assign to testers - 5 tasks per set. They shouldn't take any more than 20 minutes to run through. I think if we could get at least 5 testers per task set we'd be in good shape.
I also tacked on some introductory questions so we could get a feel for the tester's previous Fedora Community experience or their initial impressions of the site. There's also a set of closing discussion questions to get the tester's feel for the site after running through the tasks.
What do you think? Am I missing any essential functionality coverage with these tasks? Is the balance between the two task sets fair? (I tried to start both sets off with easier/quicker tasks and balance the number of more involved tasks per each.)
~m
Test Requirements: • Users must have an FAS account. Or we need to set up a test account so they can view the people search portions of the site and so the package maintenance sections of the site are meaningful. (If they own 0 packages we'll not be able to test some of the core functionality I think we want to.) • Package portions of the test are going to be way more useful if we have package maintainers as test users.
Introduction / First Impressions (All questions on front page) • Have you seen this site before? ∘ (If yes) Where did you hear about it? • Have you used this site before? If so, what have you used it for? ∘ (If yes) How did that go? ∘ (If yes) How often have you used this site? • What are your first impressions, looking at this site now? • What do you think this site is? • What would you click on first? • Any other comments you'd like to make about this site? • What do you think of this page? Where would you click first?
TASK SET I
People
- View your profile on this site. What do you think about it? ∘ Is the information accurate? Do your group memberships seem accurate?
Do your package affiliations seem accurate? 2) You've just spoken with Dennis Gilmore and he's asked you to open a ticket for him in a fedorahosted.org trac instance. In order to cc Dennis on the ticket, you'll need to figure out his FAS account name. Where would you go to look it up?
Packages 3) Do you have any unpushed updates? Where would you go to look this up? 4) How are your builds doing? Where would you go to look this up? 5) Let's say you have a friend who is a little bit behind the times - she is running Fedora 9. There's an annoying Inkscape bug that bothers her - after searching the internet for a bit you found out the bug is fixed in Inkscape version 0.47. ∘ Is Inkscape version 0.47 available in Fedora 9 for your friend? ∘ If Inkscape version 0.47 is not available for Fedora 9, can you download an SRPM here to build it for her?
TASK SET II
People
- You need help with yelp. Who's the package maintainer?
- Luke Macken is going away on vacation for a couple of weeks and he's
asked you to look over his packages while he's gone. How many packages does he have, and do any of them need attention? Where would you go to look this up?
Packages 3) How would you file a bug against NetworkManager using this system? 4) What has changed in the 'ruby' package over the last few updates? Where would you go to look this up? 5) How's the karma of your current testing updates doing? How would you check up on this in Fedora Community?
Closing • How well does Fedora Community support your package maintenance workflow? • Would you use Fedora Community to help maintain your packages? Which parts? • Is there any essential functionality missing from Fedora Community that would help you maintain your packages more easily? • Did you learn anything about Fedora Community's functionality through this test? • What do you think is the most useful function of Fedora Community? • What are the least useful functions of Fedora Community?
Rejected Tasks (I came up with these but decided they were less important than the ones I selected)
• How many bugs are open against python right now? • You just found out from an artist friend that there's a handful of supplemental plugin packages for the Gimp that let you do cool things. Look them up - what are they? • You see someone in freenode IRC with the nickname 'halfline' and you're not sure what his/her real name is. Can you try looking it up here? • You'd like to meet with Kushal Das to discuss one of his packages, but he's in a different time zone so you're not sure when would be a reasonable time. Can you figure out your time difference with him and work out a good time to propose a meeting with him on IRC?
This looks great, Mo. With each new Fedora Community milestone, I'd like us to define various tasks that people can perform with the new features, so we can keep expanding upon these.
luke