Asking Questions about Worlds

This week, I fixed some small bugs to get the Code Tests up and running again, and then began going through all the badge requirements to look at how the current Code Tests system would be able to handle them. Through this, it became clear that the badge requirements ask several different categories of questions about Worlds:

What is in this World?

Although initially I thought the Code Tests would be bad at answering these questions, generally they are pretty good. Code Tests can figure out what characters have been added to a world and whether characters are props are grouped together fairly easily.

How does this World work?

Basically, this question is asking about what code is being used to make the World functional. A mixed bag on how Code Tests can handle these types of questions. Some it can answer very easily, such as whether the World is taking input from the user, whether there are custom actions being used, and if groups perform their actions together. Other questions are more difficult for Code Tests to answer, like figuring out which character performs a certain action.

What is this World like?

These questions have to do with descriptions of the World that are hard for a computer to make a judgement on. For example, one badge requires the user to distinguish between the visible and invisible properties of a character. This is easy for a person to determine, but hard for a computer, unless you go one-by-one and label every property as either "visible" or "invisible." As might be imagined, Code Tests currently can't answer these questions.

How was this World made?

These are questions about the steps a user took to put the World together. For example, when the user was in the Scene Setup screen, did they change the color of the sky? What about the position of the camera? While these choices affect the code, right now the Code Tests can't tell the difference between when a user takes these specific actions, or when they take different actions that accomplish the same things.

Having these four different types of questions, it is easier to see where the Code Tests are ready for prime-time, and where they still need work. In the future, I'll be looking for solutions to modify Code Tests in these unprepared areas and get the system ready to tackle all of the badge requirements.