Searching for Patterns in Mentors' Tests

My current goal for the Code Tests is to figure out what direction the system needs to move to be able to return relevant code sections consisting of multiple statements. To begin this, I spent the week going through the tests that were written by potential mentors for Michelle's study.

For each test, I took the colloquial test description and entered it into a table. If any tests were exact duplicates with different wording, I only entered them once. Then, I removed any tests that were obviously out of the scope of what code tests are capable of (for example, tests that relied on information only available at runtime, or tests that relied on subjective story information, such as who the "main character" is). In the end, I was left with just under 50 unique tests.

Then, for each of these tests, I wrote a plain-English explanation of how this test would be coded. I made sure to not include any specifics from the current code test system; it was more similar to the way that Dennis coded some example tests from Michelle's study just as Java classes. In the process, it was clear that my diagramming of the AST structure had been helpful in my understanding. For example, when investigating a "move" procedure, to determine what character is actually moving requires knowledge of four different node types: ExpressionStatement, MethodInvocation, FieldAccess, and NamedUserType; and understanding how these four relate to each other is crucial.

Clear patterns have become apparent in this process, although I haven't had the time to fully analyze all fifty entries to qualify exactly what all of them are, but I already have some inclination of what to do for code tests. Currently, there is one crawler type (ConstructCrawler) which is customizable in which node type it accepts. It seems that one way to change the code test system is to offer several different types of crawlers, with unique customization options for each.

For example, one pattern that has already emerged would dictate the need of something like a "DoubleCrawler." Several tests need to look for statements with bodies (e.g. DoTogethers), and then look the content of the bodies to see if they pass certain criterea. It would make sense then to offer a crawler type that first crawls for a construct with a body, and then crawls that body, and does its post-processing on the nodes found there. This crawler would have two customization options: "Construct With Body" to crawl for initially, and "Contained Construct" to look for once a body have been found.

There are also some clear API functions that have emerged: common processes that are needed for several tests, but might be hard for a user to write. One example would be a function that looks for a node's nearest parent Statement node. Statements actually tie code to a distinct place in the program, so if, for example, a test is concerned with a Double parameter that is used several times throughout a program, it's going to be more useful for a user to see a list of the statements that use that Double, as opposed to a list with the number "17" a dozen times.