A Proposal for Testing Badges

This week, I finished up with the code tests for the two badges I want to test and took a look at what needs to get done in order to do some testing. I thought about what kind of experience I wanted to provide for the user, especially in light of the time constraints involved in testing at the science center. My overall goal with badges is to motivate and direct the user’s learning. However, there is a lot from my original grand plan of badges that is still unfinished. Here is a breakdown of how I am attempting to achieve that goal. The bold points are things that I have already implemented. The italicized points are things I think will be relatively easy to mock up/implement. How I’m trying to motivate: Make the connection between earning a badge and improving your story organizing badges around storytelling categories adding descriptions for each level Making badges seem “cool” placing badges on user/world thumbnails, show page and news feed highlight “top badge earners” emphasize their followee’s badges Award the user for earning the badge notifications when they earn the badge on the website notifications in the ide How I’m trying to direct learning: Giving the users specific tasks to accomplish at progressively more difficulty levels levels layout view on the badges index page (for two badges) highlight each skill/level as they earn it, so the user can see their progress have the “remix” ribbons as a step to earning the full badge (implemented on the website, but not yet testable) Suggest new badges have a notification pointing to the badges page when they first log in suggest badges to earn based on the skills/levels./badges they have achieved thus far award remix ribbon after user has remixed a skill which may inspire them to learn it on their own/learn similar ones Given what I have built/think I can build relatively quickly, here are some questions I would like to ask: Are the users motivated to earn the badges when they first encounter the site. If so, what is the main motivating factor (coolness factor, wanting to show off, storytelling goals)? If not, why? Am I effectively communicating the badges story on the site? Are the users continually motivated to earn badges once they have gotten a skill/level/badge? Why or why not? How do the users feel about the badge-earning process? (Overwhelming/too difficult, within their grasp, etc.) How do the worlds of the users who earn badges look in comparison to those who don’t? (Are the former worlds more complex? Do the users continue to use the skills? Do the non-badge earning users come across those skills anyway? Does having badges as a presence on the site motivate the users to build more worlds? How well do the skill code tests I have written hold up to a large group of worlds? Are there any obvious bugs or edge cases that I didn’t account for? Given the testing atmosphere at the Science Center (pulling in participants for small chunks of time) I think the easiest questions to look at are numbers 1 and 2. I could have the users browse the website under a “new user” perspective. They would be greeted with a notification encouraging them to earn the Stories 101 badge and see badges across the website. I could see whether or not they decide to build a world in order to earn a badge. I ask them to narrate their experience with the site, which would give me a clue about how the badges story is being read. At the end of their experience, I could have them fill out a questionnaire asking about their motivation (or lack thereof) to earn badges. In order to not influence their decisions, I would not tell them that I was specifically looking at badges. Time permitting, it would also be nice to be able to follow the badges process into the actual building a world and earning skills/badges stage. The Stories 101 skills are not difficult (sharing a world, say, etc) so this is feasible. Again, I could have them narrate their thoughts during the process and have them question their motivations at the end. Informing the users that we would like them to earn badges would increase the likelihood that they would reach this point. However, not telling them this, but allowing them to build worlds could give us some comparison points on worlds that are built for the purpose of earning a badge versus not. (Alternatively, I could perhaps compare the world that people build in order to earn a badge with the worlds that first time users have built in the past). I think this could start to get at questions 3-5 as well as give the skill/badge awarding system a nice workout. In order to fully tests questions 3-6, I think it would be really cool to have two versions of the website - one with badges and one without - and compare the worlds of the two groups over a period of time. However, I also realize that time is a limiting factor and this is therefore not feasible. As for question 7, I think this is something that can be tested by running the skills tests on the worlds currently on the website and seeing what happens. With all of these things in mind, here is my to-do list: Absolutely Necessary codetests Saving out/storing the skills test Running the tests on the uploaded worlds Correctly awarding the badges to the user community Making sure I have enough sample worlds placing badges around the site having notifications when you earn a badge/first join Would Be Nice Knowing where code comes from (challenge, remix, etc) This affects the badges paradigm in terms of first awarding the remix level and then working your way up, however we can still test other parts of the badges system without the remix It also affects my ability to award the skills “open a challenge” and “remix,” the latter makes up Stories 101 level 3 by itself. Potentially, I could just make up another skill or go in and award this skill manually if I see that they have remixed something. I could also just do away with this level altogether since I only have two badges running. Knowing what the user is doing (play, zoom) Having the user play their world, while nice, is not an essential skill at this point and its level can exist without it Zoom is the essential part of Stories 101 level 4. As with remix, I could award this skill manually, replace it with something, or just do away with the level. If you’ve made it this far in this extremely long blog entry - congratulations! Also, here are some things that I would love feedback on: My proposed testing plans in terms of ability to answer my questions/implementations. Any comments on feasibility. (Am I trying to tackle way the heck too many things at once?) The importance of testing remix at this stage, especially considering this isn’t something that hasn’t been implemented in code tests yet.