A7: User testing and redesign – DUE Tuesday, Nov. 27 @ 12:30am (note unusual date!)
The goal of this assignment is to test your app prototype with two people to further streamline your app and to inform the online A/B testing that you will do for A8.
Step 1: In-person user testing
Observe at least two different people testing your app prototype in-person. Try your best to find representative testers whom you would expect to use your app in real life; if you cannot do so, then get as close as possible to approximating your target population.
One person will facilitate the test and interact with the user, and the rest of the team will be in charge of taking notes/photos/video/etc. The facilitator should use the protocol that you developed in A6. Don't forget to have your user sign a printed consent form before beginning the test (some user testing may be unethical without informed consent). Unlike in A3's heuristic evaluations, this time around your user will not be writing down the problems they find for you. It's your job to learn what the people testing your prototype are thinking, and what mental models they are forming; the feedback they provide you will be very valuable for your next iteration. Your main goal here is to find ways to improve your interface.
Submit your testing protocol and signed consent form for each participant. Immediately after each test, do a quick debrief with your team and write down any reactions or thoughts that came up during the testing session.
Take a photo or draw a sketch of each tester using your prototype. As with the A1 needfinding assignment, these photos with captions should show breakdowns and design opportunities. Contextualize them by capturing the action, e.g. by using over-the-shoulder photos to capture the user using your app, and by showing the setting. Look for other breakdowns and pain points in your interface and try to understand what the problems are and how you might fix them. When possible, modify/update your app prototype before testing on the next participant so that you can get fresher data.
Watch the In-Person Experiments lecture video for some practical tips about running these sorts of experiments.
Step 2: Compile your findings
After testing, take some time with your team to reflect on your findings. Discuss as a team and define some general patterns in people's behavior. When you identify some interesting points, talk deeply about them—ask each other questions, recreate the different tests, analyze the decisions people made, other paths they could have taken, and so on.
Submit a detailed and understandable list of changes that you will implement as a result of your testing and discussion, with justifications. Fix the bugs that are either small and easy to fix, or too severe to ignore. Make sure that you do this before moving on to the next step of this assignment.
Step 3: Create a Meaningful Redesign
Choose ONE non-trivial component of your prototype to redesign in order to either resolve a breakdown or provide a potentially better solution than what was created before, as informed by user testing. To do this, make a duplicate of the specific webpage where the change will take place and change the URL route. For example, if you are redesigning a component of the homepage, keep the current homepage ("homepage.htm") and create a second page with a route "homepage2.htm".
Do not overwrite the current version of the page and do not create an entirely new repository for this redesign! You will be submitting both the original URL and the redesign URL.
The redesign will be used for online A/B testing in the next assignment (A8), which requires the use of the chi-squared statistical test. Therefore, the aspect you choose to redesign must be testable using the chi-squared test, which usually tests an aspect that can be binarily classified (i.e. user clicked or didn't click a button). The redesigned component must also be noticeably different from the original design in a substantial way—just changing the background color as your redesign is not enough. We highly suggest that you read A8 in detail to understand what kind of redesign can be measured by chi-squared tests during online A/B testing. See the lecture video on "comparing rates" for more information.
Step 4: Description of Online A/B Test
Submit a description of the online A/B test that you plan to run for next week's assignment (A8); you do NOT need to run the online A/B test yet. The description should explain what the differences between your original and redesigned versions of the component are (from Step 3), and how the results of these changes will be measured in your online A/B test next week. Also include your best guess of all possible outcomes and interpretations for next week's test, as interpreted with implications on the design of the prototype.
Step 5: Update Development Plan
Just as we've been doing in the previous weeks, update your development plan. Add new tasks for this week and the following weeks while marking when existing tasks have been completed. Add stretch goals that you find feasible and adjust other tasks that may be out of reach.
Here are three student examples (note that the assignment was probably different in prior years):
Submit a single well-formatted PDF file for your entire team with the following items concatenated within it:
Submit your single formatted PDF in Gradescope to the bin for your studio section. Only one team member needs to submit on behalf of the entire team.
Evaluation criteria & Grading rubric
The rubric below contains criteria that are worth one point each and will be graded independently and in a binary fashion.
Due In Studio: Teammate Assessment