A8: Comparative Evaluation and Finalize – DUE Friday, Dec. 7 @ 12:30am

You will perform an in-person comparative evaluation that you planned last week. Then you will finalize your app and be finished for the quarter!

Step 1: Perform comparative evaluations

Each team member needs to find at least 3 people on which to test your original and redesigned component in person; for a 3-person team, that means 9 total test participants. (We will give you time in class on Monday, Dec 3 to do these tests.) You can split up the work however you want, as long as you get the total required number of participants; e.g., you can test all 9 together as a team, or split it up in any other way.

Unlike the whole-app user testing for A7, for this test, you will perform a comparative evaluation to compare user experiences with your original and redesigned single component. This test should be much faster since the tester needs to only try out those 2 HTML webpages instead of going through your entire app. (You also do not need to submit a signed consent form this time.)

Each user should interact with both versions: the original and the redesigned webpage. Also, as the test facilitator do not tell them which is the original and which is the redesigned one, to eliminate possible biases. Randomly assign the order that each user sees the two pages; some users should see the original first, others should see the redesign first.

You should observe what the user is doing when interacting with both versions of your component, and measure either quantitative or qualitative data that is relevant to the task you want them to perform. For instance:

  • quantitative data could include how frequently the user performed certain actions on the page, such as clicking on buttons, scrolling, or interacting with page elements.
  • qualitative data could include what kinds of feedback the user says out loud as they are thinking aloud during the study, or what kind of errors or breakdowns that occur while they are testing it.

(If you get the user's permission, you can record the screen to review the data later, but it may be easier just to take notes while you observe. You don't need to turn in user test recordings. If you want to screen-record, see the video section for free software.)

At the end of each evaluation session, ask the user what they thought about both versions to collect comparative feedback.

We estimate that each session should take ~15 minutes.

Step 2: Analyze Comparative Evaluation Results

Collect results from your comparative evaluations and prepare a write-up analyzing your data. You should present any mathematical analysis that you performed for quantitative measures, or a thoughtful written analysis of your qualitative measures, along with any insights or discoveries that your analysis revealed. Draw some meaningful conclusions about your results by interpreting what they might mean for your app's final design.

Refer to the rubric for what your analysis will be graded on.

Step 3: List of Revisions

What did you learn from these comparative evaluations? Compile a list of revisions or improvements you would like to implement based off the results. Additionally, include other revisions that your team would like to make. (You may not have time to implement all of these by the end of the quarter, but we want you to at least think about possible revisions.)

Step 4: Finish Aesthetic Details

Focus on fixing the interaction of your app based off your comparative evaluation results. Also, now is finally the time to make it look good! You don't necessarily need fancy professional-looking visual design; you should just aim to make your app's UI look polished, which means that it looks complete and not like a work-in-progress.

By this final deadline, your app should be completely done, both in interaction and aesthetic design.

Step 5: Brand Yourself

Create a finalized app name that encompasses the function and brands the app. Branding is an important part of showing uniqueness and attracting users!

Please pick a family-friendly and tasteful name, since this finalized app name will be used to introduce your team during final presentations.

Step 6: Create a 1.5-minute Demo Video

Create a 1.5-minute demo video showing your app in action with voice narration, which will be played during final presentations during the final week of the quarter. Keep it short, while showing off your point of view, project concept, and all the main features of your application. Record the screen of your web app itself with free screen recording software such as Quicktime on Mac, Camtasia (free trial for Windows and Mac), or using your phone's camera (but make sure everything is clearly visible).

(We will be strict about enforcing a 1.5-minute time limit for these videos, to be fair to all students. Think of this like a hard 'page limit' on written assignments.)

Refer to the rubric to find out what to include in your video, since that is what we will be grading you on.

Your TA will be grading your demo video, and you will be showing it off in front of the class too, so make something you are proud to show off!

Assignment Submission

Submit a single well-formatted PDF file for your entire team with the following items concatenated within it:

  • Names and PIDs of all of your team members, along with your team name. (If you forget someone's name, they will not get credit for this assignment.)
  • The data that you collected. Organize this information so that we can get a sense of the raw data that you gathered. What feedback (quantitative or qualitative) did you receive? We recommend organizing this information into a table or spreadsheet, exported as PDF. (Comparative Evaluation Results)
  • Summary of your findings. You should present any mathematical analysis that you performed for quantitative measures, or a thoughtful written analysis of your qualitative measures, along with any insights or discoveries that your analysis revealed. Draw some meaningful conclusions about your results by interpreting what they might mean for your app's final design. (Analyze Comparative Evaluation Results)
  • List of the changes you would make as a result of your comparative evaluation. We understand that your test may have concluded only recently, and you would have had time to make only small changes. But we would like you to think about additional features or improvements that are motivated by your results. (List of Revisions)
  • Your final app name. This is the name that will be used in your final presentation. (Brand Yourself)
  • The URL of your final app. Do not forget to give us login information. Note: the URL must work at least until your assignment is graded. If it doesn't work, you'll receive no credit. Very important: the contents of this URL should not change in the upcoming week while your TA is grading this assignment, or else that is a violation of the academic honesty policy (Finish Aesthetic Details)
  • URL link to your 1.5-minute final presentation video (Presentation video URL)

Note: since we may grade your assignment up to a few days after submission, per the honor code, we expect that the prototype URL show the state of your prototype at the time of submission.

Submit your single formatted PDF in Gradescope to the bin for your studio section. Only one team member needs to submit on behalf of the entire team.

Evaluation criteria & Grading rubric

The rubric below contains criteria that are worth one point each and will be graded independently and in a binary fashion.

  1. Evidence is presented that the comparative evaluation was performed on at least 9 users (assuming a team of size 3 -- if your team is larger or smaller, scale it accordingly), by showing a table or spreadsheet summarizing information about each user such as their name, occupation, and raw collected data from their test session (e.g., number of event occurrences or qualitative observations).
  2. Analysis of your measures is presented, either as math/graphs for quantitative measures or as a detailed descriptive write-up comparing qualitative measures. (2 points)
  3. Insights and conclusions drawn from your measures is written and is based off the results of your analysis.
  4. List of revisions is based off the comparative evaluation or other feedback that your teammates gathered.
  5. App has a clear interaction flow that makes usability understandable to new users.
  6. Functionality of app is complete and free of bugs. (2 points)
  7. App interface is aesthetically polished (i.e., looks complete and not still like a work-in-progress). (2 points)
  8. App follows an aesthetic design that is consistent across webpages.
  9. App has a clean look; words/buttons are not cluttered and information is not presented in such large chunks that they can be overwhelming to a user.
  10. App contains text that is easy to read (i.e. no white text on light background or vice versa) on all webpages.
  11. Link to demo video is publicly viewable, and the video is 1.5-minutes long (no credit if it is longer than 1.5 minutes).
  12. Demo video communicates the app at a high level without seeming cluttered.
  13. Demo video sufficiently explains the point of view and project concept with relation to the class's design theme.
  14. Demo video clearly articulates a user need that the app aims to address.
  15. Demo video shows a walkthrough through the important functionality of the actual app itself working in a web browser.

Due In Studio: Teammate Assessment

During studio, click here to assess how each of your teammates contributed to this assignment.