Saving Time and Sanity Using Dapple’s Next-Generation Testing Tools

Throughout 2016, an increasingly recurrent topic in the web developer community has been "JavaScript fatigue" — a weariness with the exponential rate of change in what's considered to be best practice tools, frameworks and principles for writing JavaScript applications. Text after text has been written about this, to the point where people are starting to become fatigued with the whole "JavaScript fatigue" meme.

While the sheer existence of an overabundance of developer tools may be a serious issue facing the JavaScript community, the problems we have as Ethereum developers are more like the direct opposite of this. We need a lot more high-quality tools, frameworks and best practices.

In fact, if "Ethereum fatigue" were a thing, it would probably be a name for the feeling you get when your code is crashing with a "bad jump destination" or "out of gas error" and you take a deep breath as you prepare for a long night of literally commenting out large sections of your code at random.

Part of our mission at Nexus Development is to make Ethereum development a pleasant, ergonomic and delightful experience. We've come a small part of the way with Dapple and Dappsys, but we're really just getting started.

Today, we're offering a sneak peek at the next version of our pure-EVM testing infrastructure and one of its new features — namely, stack traces.

First, let's look at a very simple example:

import "dapple/test.sol";

contract FooTest is Test {
    function test_foo() {
        this.foo();
    }

    function foo() {
        throw;
    }
}

Here's how Dapple currently reports this test failure:

$ dapple test
Testing...

FooTest
  test foo
  Error: VM Exception while executing transaction: out of gas

Failing Tests
  FooTest: test foo

Summary
  Failed 1 out of 1 tests.

With the new tooling, here's how it looks instead:

$ dapple-quicktest
Building...
Trying FooTest...
F

  [CRASH] FooTest.test_foo
            Dapple -> <FooTest#d289> .test_foo() => BadJumpDestination
    <FooTest#d289> ->   <FooTest#d289> .foo() => BadJumpDestination

0 tests passed, 1 failed
Took 0.266s

We can see that the test failed because the foo method (which is called by test_foo), caused a BadJumpDestination error. Pretty nice.

Now let's look at a more complicated example. We'll use ds-hello (an example project demonstrating some basic Dappsys features).

~/src$ git clone https://github.com/nexusdev/ds-hello
~/src$ cd ds-hello

First, we'll check that everything works:

~/src/ds-hello$ dapple test
Testing...

DSHello50Test
  test greeting
  Passed!

Summary
  Passed all 3 tests!

Okay. Now let's introduce a bug. Here's an excerpt from the test class:

function setUp() {
    database.setAuthority(root);
    controller.setAuthority(root);
    controller.setDatabase(database);
    frontend.setAuthority(root);
    frontend.setController(controller);

    root.permit(controller, database, "set(bytes32,bytes32)");
    root.permit(frontend, controller, "greet(address)");
}

function test_greeting() {
    controller.setGreeting(bob, "hello world");
    assertEq32(bob.greet(frontend), "hello world");
}

Hmm... what if we forgot to give the controller access to the database?

    //root.permit(controller, database, "set(bytes32,bytes32)");

Let's see...

~/src/ds-hello*$ dapple test
Testing...

DSHello50Test
  test greeting
  Error: VM Exception while executing transaction: out of gas

Failing Tests
  DSHello50Test: test greeting

Summary
  Failed 1 out of 3 tests.

No clue as to where the problem is. Ethereum fatigue already starting to set in.

Of course, to make matters worse, you can imagine that you noticed this failure only later — perhaps because you made this change accidentally or maybe because you did it as part of a large and messy refactoring. The point is, if you can't remember what you did to break this test, you're going to have to roll up your sleeves and start dividing and conquering.

But now check this out:

~/src/ds-hello$ dapple-quicktest
Building...
Trying DSHello50Test...
..F

  [CRASH] test_greeting
                        Dapple -> <DSHello50Test#d289> .test_greeting() => BadJumpDestination
          <DSHello50Test#d289> ->   <DSHello50Controller#01a3> .setGreeting(<FakePerson#8932>, 0x68656c6c6f20776f726c64000000000000000000000000000000000000000000) => BadJumpDestination
    <DSHello50Controller#01a3> ->     <DSMap40#b48c> .set(0x5ed9b03c6b9d04e95559c0898a646253f0fb59deeaf8d9d92f138836db16e50b, 0x68656c6c6f20776f726c64000000000000000000000000000000000000000000) => BadJumpDestination
                <DSMap40#b48c> ->       <DSAuthority50#f0b2> .canCall(<DSHello50Controller#01a3>, <DSMap40#b48c>, bytes4(keccak("set(bytes32,bytes32)"))) => (false)

2 tests passed, 1 failed
Took 0.308s

Here, we can immediately see that calling <DSMap40#b48c>.set from <DSHello50Controller#01a3> caused a BadJumpDestination, which happened after <DSAuthority50#f0b2>.canCall returned false — which indeed points to a controller–database authorization problem.

What's especially cool here is that we get to see the arguments and return values of each call. The test framework is even smart enough to do some tricks like helpfully showing the preimage of a signature hash appearing in a bytes4 value (as seen in the third argument of canCall).

Well, you can see how useful this is and how much time it can save.

While it's not quite ready to be the default dapple test, this is under active development and is already being used internally at Nexus and MakerDAO.

If you too would like to use these features today, first make sure you have an up-to-date installation of Rust. Then, use the following commands to download and install ethrun and dapple-quicktest:

git clone https://github.com/nexusdev/ethrun
make link -C ethrun
git clone https://github.com/nexusdev/dapple-quicktest
make link -C dapple-quicktest

At this point you should be able to start using dapple-quicktest as a drop-in replacement for dapple test. If you have problems with the installation or otherwise, feel free to ask for help in our Rocket Chat.

(The ethrun utility allows dapple-quicktest to execute the tests using the Parity engine. By the way, if your tests are gas-heavy, or if you have a large number of them, this could speed up your test runs by 100x.)

Happy hacking, and stay tuned for more developer experience upgrades from Nexus as we go into 2017.