A8: A/B Testing and Finalize – DUE Friday, Dec. 7 @ 12:30am
You are going to be using Google Analytics to run the online A/B test that you planned last week. Then you will finalize your app for this final assignment.
We are giving you a bit more time for this final assignment because it is longer than the rest, so please do not wait until the last minute.
Step 1: Launch Online A/B Test
Set up your A/B test with Google Analytics; refer to Lab 5 for help. Since you will be statistically analyzing the results, recruit at least 20 users. Put another way, that means that each person on your team should recruit at least 7 people each: a couple of dorm-mates and a few family members, and you're basically there. (Thanksgiving break is a great time to get relatives at home to try your app!)
For simplicity, not all of your test users need to be in the target population for the design theme; try to get as close as you can, but we recognize that it may be challenging to do so in a short time span.
You can also use social media like Facebook and Twitter to recruit online testers. You can even recruit other people in class. Make sure to launch your test several days before the deadline, though, since it will take time to collect and analyze your data. Here are some ideas from Optimizely for A/B testing.
Google Analytics takes at least 24 hours to report back data to you, so do not start this part last-minute! If you start this last-minute, then you will not be able to collect data in time to finish the assignment. Also, your Google Analytics setup code may have bugs, so you should start early and leave plenty of time to debug.
Step 2: Analyze A/B Test Data
Collect your results from your A/B test, which should be comparing the number of users who, for instance, either clicked through a navigational funnel or didn't, viewed a particular page or didn't, or pressed a button or didn't, in both your A and B conditions. Present your data and the results: show the inputs to your chi-squared test, your calculations, the resulting chi-squared value, and the p value (since you won't have that many users, we don't expect everyone to achieve p < 0.05 statistical significance, so don't worry if you don't). Tables of both your observed and expected values are the best way to visualize your data and calculations. This online tool is a helpful resource for calculating the chi-squared statistic. However, you must show your work for your calculations just like we showed in lecture. Watch this video to see how it's done. Remember that the chi-squared test compares numbers of users, not times or durations. If you find yourself using a different measure, e.g. time spent on a page, then find the appropriate statistical test to show your calculations and interpret its results. The TAs will not help with anything other than the chi-squared test, so do so at your own risk. Thus, we highly recommend you measure something that can be tested with a chi-squared test.
Next, discuss your findings in a very brief paragraph: can you draw solid conclusions or are additional experiments needed? What changes would you make based on these results? What other things would you test in the future?
Step 3: List of Revisions
What did you learn from these tests? Compile a list of revisions or improvements you would like to implement based off the results of the A/B tests. Additionally, include other revisions that your team would like to make that may not have been discovered through the user tests. (You may not have time to implement all of these by the end of the quarter, but we want you to at least think about possible revisions.)
Step 4: Finish Aesthetic Details
Focus on fixing the interaction of your app based off your A/B results. Also, now is finally the time to make it look good! Pay attention to detail and ensure that the app is optimized for a mobile interface (if you're targeting mobile). By this deadline, your app should be completely done, both in interaction and aesthetic design.
Step 5: Brand Yourself
Create a finalized app name that encompasses the function and brands the app. Branding is an important part of showing uniqueness and attracting users!
Please pick a family-friendly and tasteful name, since this finalized app name will be used to introduce your team during final presentations.
Step 6: Create a 1.5-minute Demo Video
Create a 1.5-minute demo video showing your app in action with voice narration, which will be played during final presentations during the final week of the quarter. Keep it short, while showing off your point of view, project concept, and all the main features of your application. Record the screen of your app itself in a web browser with screen recording software such as Quicktime on Mac or Camtasia (free limited-time trial download).
(We will be very strict about enforcing a 1.5-minute time limit for these videos, to be fair to all students. Think of this like a hard 'page limit' on written assignments; it is not fair to your classmates if you go over the allotted time, since we have only limited time in class to play the videos.)
Refer to the rubric to find out what to include in your video, since that is what we will be grading you on.
Your TA will be grading your demo video, and you will be showing it off in front of the class too, so make something you are proud to show off!
Submit a single well-formatted PDF file for your entire team with the following items concatenated within it:
Note: since we may grade your assignment up to a few days after submission, per the honor code, we expect that the prototype URL show the state of your prototype at the time of submission. You will very likely be updating your prototype after submission, but please do so on a separate URL.
Submit your single formatted PDF in Gradescope to the bin for your studio section. Only one team member needs to submit on behalf of the entire team.
Evaluation criteria & Grading rubric
The rubric below contains criteria that are worth one point each and will be graded independently and in a binary fashion.
Due In Studio: Teammate Assessment