A7: User testing and redesign – DUE Tuesday, Nov. 27 @ 12:30am (note unusual date!)
The goal of this assignment is to test your app prototype with two people to further streamline your app and to inform the comparative evaluation that you will do for A8.
Step 1: In-person user testing
Observe at least two different people testing your app prototype in-person. Try your best to find representative testers whom you would expect to use your app in real life; if you cannot do so, then get as close as possible to approximating your target population.
One person will facilitate the test and interact with the user, and the rest of the team will be in charge of taking notes/photos/video/etc. The facilitator should use the protocol that you developed in A6. Don't forget to have your user sign a printed consent form before beginning the test. Unlike in A3's heuristic evaluations, this time around your user will not be writing down the problems they find for you. It's your job to learn what the people testing your prototype are thinking, and what mental models they are forming. Your main goal here is to find ways to improve your interface.
Immediately after each test, do a quick debrief with your team and write down any reactions or thoughts that came up during the testing session.
Take a photo or draw a sketch of each tester using your prototype. As with the A1 needfinding assignment, these photos with captions should show breakdowns and design opportunities. Contextualize them by capturing the action, e.g. by using over-the-shoulder photos to capture the user using your app, and by showing the setting. Look for other breakdowns and pain points in your interface and try to understand what the problems are and how you might fix them. When possible, modify your app prototype before testing on the next participant so that you can get fresher data.
Watch the In-Person Experiments lecture video for some practical tips about running these sorts of experiments.
(You'll have time in class on Monday, Nov 26 for user testing, but we highly suggest that you get it done earlier since it's due the next day, and there are many other parts of this assignment.)
Step 2: Compile your findings
After testing, take some time with your team to reflect on your findings. Discuss as a team and define some general patterns in people's behavior. When you identify some interesting points, talk deeply about them—ask each other questions, recreate the different tests, analyze the decisions people made, other paths they could have taken, and so on.
Write up a detailed and understandable list of changes that you will implement as a result of your testing and discussion, with justifications. Then fix the bugs that are either small and easy to fix, or too severe to ignore. Make sure that you do this before moving on to the next step of this assignment.
Step 3: Create a Meaningful Redesign
Choose ONE non-trivial component of your prototype to redesign in order to either resolve a breakdown or provide a potentially better solution than what was created before, as informed by user testing. To do this, make a duplicate of the specific webpage where the change will take place. For example, if you are redesigning a component of the homepage, keep the current homepage ("homepage.htm") and create a second page with a route "homepage2.htm". You will be submitting both the original URL and the redesign URL.
The redesign will be used for comparative evaluation in the next assignment (A8). The redesigned component must also be noticeably different from the original design in a substantial way—just changing the background color or some simple visual features as your redesign is not enough. We highly suggest that you read A8 in detail to understand what kind of redesign is appropriate.
Step 4: Description of Comparative Evaluation
Submit a description of the comparative evaluation that you plan to run for next week's assignment (A8); you do NOT need to run the actual evaluation yet. The description should explain what the differences between your original and redesigned versions of the component are (from Step 3).
Next you should write down what you plan to measure in your comparative evaluation for A8. You can either choose to use quantitative or qualitative measures. For instance, you can count how frequently users perform certain actions in your app (quantitative), or what types of comments they provide while thinking aloud (qualitative) when using both your versions. You don't need to be too formal here, and you can change your measures once you start doing A8 and see what works and doesn't work in practice; we just want you to propose a realistic-sounding plan at this point.
Finally, include your best guess of all possible outcomes and interpretations for next week's comparative evaluation, and what that may mean for the design of your final project. In other words, put down what you think will happen next week.
Step 5: Update Development Plan
Just as we've been doing in the previous weeks, update your development plan. Add new tasks for this week and the following weeks while marking when existing tasks have been completed. Add stretch goals that you find feasible and adjust other tasks that may be out of reach.
Here are three student examples. However, the A7 prompt from prior years was different in that they could make paper prototype redesigns, whereas we want you to implement your redesign in code. Also you will do an in-person comparative evaluation instead of an online A/B test.
Submit a single well-formatted PDF file for your entire team with the following items concatenated within it:
Submit your single formatted PDF in Gradescope to the bin for your studio section. Only one team member needs to submit on behalf of the entire team.
Evaluation criteria & Grading rubric
The rubric below contains criteria that are worth one point each and will be graded independently and in a binary fashion.
Due In Studio: Teammate Assessment