Unit Tests and Headless ExecutionThis week I continued to work on preparing to take the Code Test code to the server. I believe that now, we are arguably in a good place to begin working on that (more about "arguably" in a second).
The main change I made that now allows Code Tests to run (again) on the server was to make the code test execution, once again, headless. One of the changes I had introduced to the runtime was feedback for the user when they're writing tests in the IDE. This was for three reasons:
- It allowed the user to include print statements in their Jython code for debugging purposes
- When there was a Python/Jython error, it allowed the user to see the error messages and correct their tests.
- It allowed feedback on the state of the test ("Ready", "Running...", "Done", or "Failed"). This prevented a scenario where a test crashed from being identical to a scenario when a test ran but found no results.
In addition, I also rounded out a set of 19 unit tests that check Code Tests for expected execution of correct tests, and expectedly interrupted execution for incorrect or dangerous tests. The good news is that 18 out of these 19 tests are running perfectly, exactly as expected.
The not awful news is that the one test that's failing, although a very crucial test--that users aren't allowed to read/write to files on the server--is failing because the preLaunch() utility function isn't about to execute from the JUnit tests, and this is what sets up the permissions and security manager. When trying to read/write to files from the IDE code tests window, behavior fails as expected (because preLaunch() has run). I had a hard time trying to get preLaunch() to run from the JUnit tests, I believe because JUnit tests have their own permissions that limit what can be accessed, although I'm not positive that's what's happening. So the bottom line is--if we can accept that the SecurityManager unit test fails because preLaunch() isn't running, but in situations where preLaunch() does run, that test would pass, I feel like we're in good shape to start turning our attention server-side.