Teddy Billings
Contact Resume About Work
CLOUD MACHINE
A lower/middle grades fractions puzzle game.
Process Overview
EMPATHIZE Stakeholder and SME interview
ANALYZECompetitive insights Open team brainstorm Constrained team brainstorm
PROTOTYPE Prototype and tech exploration
VALIDATE Conextual inquiry and usability testing Incorperating test results
IMPLEMENT Design sepcifications
VALIDATE Usability and gameplay testing
IMPLEMENT Design insightsV1 stats and reflections
Cloud machine is a puzzle game created for students in grades 3-8. Students play the game by moving liquid across a sequence of gates. They split liquid (divide fractions) mix colors (add fractions) and measure the liquid (identify fractions) in various ways.
DATES
April 2019 to Present
TOOLS
Adobe XD, Unity, Illustrator, Splunk
ROLE
UX Designer & Researcher
TEAM
Product Manager, Dev Team, Subject Matter Experts
Empathize
Stakeholder and subject matter expert interviews

The project started off with internal stakeholder interviews with the Product manager and the Subject Matter Expert (an elementary school teacher) on ways to help teachers in the classroom. Specifically the focus was on how to better help students learn fractions.
Stakeholder key findings Teacher key findings
  • There is a large open space for our product to help teach fractions lower and middle grade students
  • Many teachers request additional fraction fluency
  • We already have a product that teaches fractions through numerals, so we want to come up with interesting ways to visualize fractions
  • We want the product to be both remedial (can help a student who is behind) and on-grade-level
  • The product should be framed as a learning game, supplemental to a lesson (not a lesson itself)
  • Some teachers will place students in small groups to do remedial activities, such as fractions
  • Many students struggle with the translation between a fraction and the number value. They can see if a pie is cut in half, but the step of identifying that as a half is difficult.
  • We already have a product that teaches fractions through numerals, so we want to come up with interesting ways to visualize fractions
  • Teachers are pressured to stick to the common core standards.
  • Teacher feel that many students are unmotivated to learn, especially if they are already behind. They think the distraction of gamification may help.
Anaylze
Competitive Insights

The next step was to look at different solutions in the design space. These offer points of inspiration for the team for a brainstorm, but also give us a sense of solutions that we don’t like, and see how we can improve upon them.
Open team brainstorm

The first (of many) brainstorms came in the form of a general open game-jam. To facilitate, we broke into small groups of 3 people each. Each team had 20 minutes to generate as many ideas as possible, with 20 additional minutes to select the best ideas and present them back to the team. We brainstormed around the following statement: How might we create a fractions math game?
Constrainted team brainstorm

After presenting all the ideas, a smaller team (design, product and SME) meet to discuss the next steps. After looking at all the ideas, there was one common theme that kept being discussed. Liquid is a great and flexible visualization tool to help understand fractions. With that in mind, we conducted another group brainstorm around the following statement: How might we use liquid to visualize fractions in a game?

The team came up with 5 ideas to prototype:
  • Ghost lines - Students measure the amount of liquid through fractions by dragging their mouse and creating lines.
  • Paint smear - Students combine paint by dragging their finger across the screen and splitting the liquid into fractions.
  • Water gate - Students move liquid by opening gates and create fractional amounts by splitting liquid and adding it back up.
  • Iterator - Students make their own fractions by splitting containers (denominator) and adding water to them (numerator).
  • Shape Clippers - Students break shapes into different sizes by cutting them and drawing lines.
An example of the "Ghost lines" non-dev wireframes to show the main interaction of measurement.
Prototype
5 prototypes & tech explorations

The team was able to build these prototypes within 2 days, with the understanding that it was throw away code. There were many objectives, but here are some of the main ones:

  1. How will students react to these prototypes? What do students like and dislike about each?
  2. How feasible is it to create this game in a limited time frame?
  3. What other opportunities will the game open up in terms of fractions and liquid?
We decided to test 3 of the 5 prototypes.

We dropped the Shape Clippers game because it was technically unfeasible with the given time frame. We also dropped the game called Paint smear, since it ended up being too limited in gameplay and would not be scalable for our needs later on. However, paint smear ended up being a critical component later on in the process, as it inspired how the game works for color mixing.
Validate
Usability testing & contextual inquiry

We wanted to show the game to our users and see their initial reaction. While the games have very poor ux and design at this stage, we wanted to make sure students were able to make the connection between the game and the math. The lack of UI and UX did cause issues with user testing, especially since students were more biased towards interfaces that were easier for them to understand, but the tradeoff is that we can learn very VERY early about what ideas will work and what ideas will not.
Show more research
Key findings

  • The visualization of the iterator was confusing, and only students with a very high knowledge of fractions were able to conceptually understand what was happening.
  • The watergate prototype was by far the most fun for most students. They also found it the most intuitive with little to no instruction. However, this prototype had the least conceptual understanding of fractions.
  • The ghostlines prototype was very difficult to use, so many students did not enjoy it. Students often used trackpads, and the interaction is dragging, which can be very difficult for computers with poor sensitivity. Once the student understood the game, they found it too easy, as you are basically just lining up the bottom and top visuals.
  • How might we incorperate the main mechanics and interaction of the Watergate prototype with the visulization of the iterator and the precision and measurement of the ghostlines without causing a difficulty spike?
Incorperating user test results

After getting feedback from our users, the next few steps were clear. We wanted to use watergate as basis for the main game; however, there were some powerful elements from other ideas that we wanted to incorporate.
The Watergate prototype became the core game loop, since it was the most scalable and most intuitive for students
Paint smear inspried the color mixing aspect, which helps with addition of fractions
The iterator inspired the pumps. We wanted students to be able to make their own fractions.
While the ghost lines interaction was difficult, the visualization was strong. We removed the interaction all together and incorperated the visuals into the main game.
Implement
Design specifications

The next (and longest step) is putting all of the user and stakeholder feedback into visuals and specifications. To do this, we used Adobe XD for interactions, and Illustrator for artstyles. We also used confluence to create a holistic design specification, and separated dev tasks using JIRA.

The objective was to create the minimum viable product to then test again with more users.
The user flow of the base game
Validate
Usability and gameplay testing

After the MVP was created, we did another round of user testing. This time, it was much more focused on specific qualitative performance indicators and research questions. We wanted to know:

  • How fun is Cloud Machine?
  • How Usable is Cloud Machine?
  • Is Cloud Machine successful at its learning objectives related to fractions?
  • What authorable content can be changed before release?
Key Findings

  • Overall, results were mostly positive, students rated the game an average of 8 on a scale of 1-10
  • Students disliked the overal difficulty of the game and wished it was easier.
  • There were many issues with the discoverability of the what gates were clickable and what gates were not.
  • “Conceptual blocks” where students reached content above grade level were frustrating
  • Most interactions were not a problem for students, and if it was a problem, they generally relied on the teacher or other classmates around them to solve some issues.
Show more research
Implement
Incorperating user test results

And with that, we made a few more changes before our v1 release. While there were more issues presented in the user test that are not talked about, we specifically wanted to focus on realistic changes that could be finished in 1 sprint before code freeze.
Add a hover state and more color so that students know what is and isn't clickable. Removed color from the unclickable lines
Change the order and add additional levels so that the game is longer and moves at a slower pace.
Add more worlds with similar art styles
Next steps & reflection

Since the last round of user tests was focused on specific changes that were realistic to make before the release, another round of user tests were conducted to better evaluate potential features to increase engagement.

Additionally, we now had quantitative data for us to sort through. The next step is to use the quantitative data to determine next steps. Particularly, many students fail levels frequently, which is an indicator of guess and check behavior. The next user test may investigate why this is happening.

In retrospect, the timeline was very tight, and the final user test was so specific in nature that much of the valuable information was unusable. However, releasing early has many benefits, and was a tradeoff well worth making in this case.
More projects
Project Imagine
Teaching WWII by letting students immerse themselves in primary source artifacts.
Assignment Viewer
Designing a scalable content viewer to let students and teachers view multi-part assignments.
Puzzle Creator
Increasing engagement for Middle-grade students by allowing them to create and share their own content.