Preliminary Evaluation of the Current ActionScript Format and Promising Avenues for Improving Remixing with Machine Learning

This week's blog post consists of a progress report on the near term goal of improving the format in which animations are saved for remixing (currently called ActionScripts), and brainstorming results for the long term goal of incorporating machine learning into the remix process.

Preliminary Evaluation of the Current ActionScript Format

I have been analyzing the current ActionScript format.  The process begins with a begin node and an end node selected from the abstract syntax tree (AST) of a world (presumably the nodes are selected through the remix interface).  A ReferencesCrawler then traverses the AST starting from the shared parent of the begin and end node, copies the AST and returns the root node of the copy.  In the process of traversing the AST, the ReferencesCrawler also saves useful information and performs corrections for errors in the selection of the beginning and end nodes.  Regardless of how we decide to store the animations, we will be using the ReferencesCrawler.

A PrecompiledScript encapsulates the ReferencesCrawler and the useful data it extracts.  The PrecompiledScript provides clues about what information obtained by the References Crawler is actually important.  The PrecompiledScript keeps track of the following:

1)The roles that need to be substituted for during remixing.

2) The script root node and the begin and end node.

3) Node field references (I don't yet know what these are for).

4) Vehicles to set (if the vehicle for an object was changed from the default vehicle).

5) Orphan parameters (objects declared outside the selected code but referenced inside the selected code).

6) Transient substitutes (I don't yet know what these are for).

7) A method for obtaining the lowest referenced member type for all objects in the selected code (used to determine suitable objects to substitute in during remixing).

8) Methods for determining whether the selection is singular and whether the root node should be copied.

The precomplied script is then written to a file in an XML format.  When the script is read back in from the file, it is stored as a CompiledScript.  The compiled script contains the following information:

1) The root method (of type UserMethod)

2) A user method holder (of type UserType)

3) A declaration hashcode to to object map (a map from Integers to AbstractDeclarations)

Presumably there is a way to convert the PrecompiledScript to the CompiledScript without first saving the PrecompiledScript to a file.  I will investigate this further.  It might be possible to serialize the fields of either the PrecompiledScript or the CompiledScript to use as the format for storing an animation.  Once serialized, the script is then in a binary format that is easy to send to the server and save in a database or file. 

I will also look into what information is actually needed for remixing.  Rather than storing all of the fields of either the PrecompiledScript or CompiledScript, it might be possible to use a different set of fields taken from the PrecompiledScript that takes advantage of the fact that the original world is also stored on the server.

Promising Avenues for Improving Remixing with Machine Learning

 As mentioned in last week's post, there are a lot of possible ways to use machine learning with remixes but not all of them make sense.  I believe the most useful option given the goals of Looking Glass and the technical requirements of machine learning algorithms is to implement a machine learned animation search feature.  The feature will require action tags to train and can tag new animations by determining the distance between a tagged animation and a new animation.  If the distance is less than some threshold, the new animation will be given the same action tag.  The challenging part will be deciding how to convert an AST into a vector that can be used in a distance metric.

Another possible feature is to recommend animations (or worlds) to users using a Netflix style recommendation system.  The drawback of a Netflix style recommendation system is that it will require users to rate animations (or worlds) on a scale (e.g. from 1 to 5) in order to train the algorithm.  The ratings do not have to be publicly visible, but there is still concern that we do not want users to rate others.  If we are willing to introduce privately visible ratings, then a recommender system would be an interesting and useful feature.