/customers/iconara.net/iconara.net/httpd.www/blog/wp-content/plugins/wp-super-cache/wp-cache-phase1.php Iconara » The lost FlexUnit documentation

The lost FlexUnit documentation

The first time I read How to use FlexUnit with FlexBuilder 2, which is about the only howto available on FlexUnit, I thought it was a bit fishy when it describes how to create a test suite from your test case in order to run the tests. I did as it described anyway, and it worked so I let it be.

For my next project, however, I decided to go to the bottom of things, because I thought that there must be a better way to do it.

Read on to see how it’s not supposed to be done, and how to do it instead.

The article does give you the basics of what you need to start testing your code, but you will have a substandard setup which will require a lot of unecessary maintenance. If you haven’t read it yet I suggest you do, because I will not describe the whole setup in this article.

How not to do it

There are a couple of problems with the article’s description on run a test case. It suggests that you create a static method which constructs a test suite from your test case, and the way to do this is to create an instance of the test case for each test method, passing the method as a parameter to the constructor. This is from the article’s test case TemperatureConverterTest:

public static function suite( ) : TestSuite {
    var ts : TestSuite = new TestSuite();

ts.addTest(new TemperatureConverterTest("testToFahrenheit"));
ts.addTest(new TemperatureConverterTest("testToCelsius"));

return ts;

}

I have a few problems with this code snipplet:

  1. The test names are passed as strings to the constructor when they could be retrieved by finding all instance methods with names starting with “test”. I have to repeat all the names of my tests, first as method names and then in the suite method. Also, the names are passed as strings, which means no compile time checking — every time you add a test you have to change that code, and if you change a name or remove a test. If you don’t your tests will fail, not because there is something wrong with them, but because they can’t be found.
  2. There is one instance of the test case, which in it self is unecessary and it also makes the methods setUp and tearDown quite pointless. If each test will run in its own instance, why should I collect my tests into test cases, why not have them as free functions or one test in each test case?
  3. It’s a static method, which means a lot of OO principles are impossible to apply. I could go on for pages about what it means, but let’s be satisfied by saying that this is a really crappy solution. As a thought experiment try to imagine the code that would collect different test cases into another test suite, it introduces the same problems same as the code above, doesn’t it?

Since this article is linked to from the official project page, and it’s one of two available articles on how to use the framework, you would expect it to be a) correct and b) explaning best practice.

Since there is no more documentation available (remember API Documentation is not Documenation), I had to resort to the code to get to the bottom of this. The weirdest thing is that the code actually contains more examples than the API reference, at least in the distribution I have.

A better way to do it

It turns out that the proper way to do it is much, much simpler. This code replaces the silliness above:

var suite : TestSuite = new TestSuite(TemperatureConverterTest);

(You also have to make sure that your test case’s constructor doesn’t take any arguments, which it shouldn’t have done in the first place).

The way the code above works is that the test suite scans the class for methods beginning with “test” and adds them to its list of tests to run. Pass it along to the test runner and watch your tests pass.

The rest of the code in the article does actually seem to work, I have no complaints about the mxml.

If you’re not interested in a rant on my opinions on the FlexUnit code, you can stop reading now. You should have the tools you need to start testing. What follows is just me being mean to the Adobe engineers, as usual.

Some notes on the FlexUnit code

I have looked through some of the FlexUnit code, and it confuses me. I can’t seem to understand why the TestCase class constructor takes a method name as parameter, I see no reason for it, and I don’t understand how it’s used. It works fine without and the part of the code I have read so far doesn’t seem to be concerned with the method name. The control of flow is passed wildly all over the place, in and out of TestCase, TestSuite, TestResult and so on, so I must say that I got tired of trying to follow.

I also found this little oddity:

private function isTestMethod( name:String ) : Boolean {
    //var pattern:RegExp = /test*/;
    //var result:Object = pattern.exec(name);
    if(name.indexOf("test",0) != 0) {
        return false;
    }
    return true;
}

It quite goes about the test backwards, may I suggest:

private function isTestMethod( name:String ) : Boolean {
    return name.indexOf("test") == 0;
}

A little easier to read perhaps. And it’s funny how they first thought that using regexps would be nice, defined a regexp that doesn’t do what they want (it matches any string which contains “tes” followed by zero or more “t”) and then in the end decided against it. That was probably just as well since they clearly don’t grasp the concept. Using regexps you could write the method like this:

private function isTestMethod( name:String ) : Boolean {
    /^test/.test(name);
}

Moving on, this code baffles me (from TestCase):

public function runWithResult( result:TestResult ) : void {
    result.run( this );
}

It may be legitimate, I’m not actually sure, but in my opinion if you name a method in imperative (such as “runWithResult”), that method should make the object (the test) do the verb (“run”) with the parameter (the result). This method does the opposite, it makes the result run the test.

36 Responses to “The lost FlexUnit documentation”

  1. akeem Says:

    At first I was confused with the documentation. I read yours and it made more sense. However some research on what flexunit was based on (JUnit) reveals that darron schall’s example is not incorrect its just the alternative method http://junit.sourceforge.net/doc/testinfected/testing.htm.

  2. Theo Says:

    Yes, you are right. I have also been in contact with the FlexUnit developers and they claim that they only do what JUnit does.

    However, that is no excuse for bad design. The static suite method is among the ugliest things I have seen in a mainstream framework. It’s completely non-OO, and the sad part is that there is a better way, but the developers themselves advertise the bad example.

  3. Holmes Says:

    Thank you, thank you, thank you. I knew there had to be a better way. I almost wrote my own damn reflection scheme to do this.

    -Jason

  4. Robert Boyd Says:

    Very useful, even if a tad harsh at times. Thanks!

  5. Thach Says:

    The xUnit framework, though its design some view as needing improvement, includes one important use case that I keep on seeing on various discussions where folks are just not willing to see. In this case, is the example that the initiator of this topic has suggested: “problems with this code snipplet” with respect to the use of test method name in statement such as ts.addTest(new TemperatureConverterTest(“testToCelsius”)).

    You must bear in mind that when you call var suite : TestSuite = new TestSuite(TemperatureConverterTest); you have no way to manage the order in which the Test* tests in your suite are to be executed. While I agree that passing string constants is not a good thing, the example that you are objecting to has merits with respect to the ordering of the tests to be executed. This is very desireable for cases where you are pushing the framework “unit” aspect towards scenario-based testing e.g. TestActionA,TestActionB, etc.. and ordering is important.

    -Thach

  6. Theo Says:

    I don’t agree. Unit tests run in isolation and the order should not be important. If the order is important you have written your tests wrong. The only dependency a test should have is what it done in setUp.

    I can see that you would want them to run in a particular order for perhaps aesthetical reasons, but I don’t find that very convincing.

    Even if testB fails before testA, testA will be run. If the order of the report is important, make your report view display the tests in alphabetical order and name the tests with an initial letter or number. Then the tests can run in whichever order that they happen to run but in the report they will look like they ran in the desired order.

  7. Steve Says:

    Likely experienced users can figure out what’s missing with the offered alternative but worthless to us noobs2testing.

    Merely replacing: var ts : TestSuite = new TestSuite();

    ts.addTest(new TemperatureConverterTest(“testToFahrenheit”)); ts.addTest(new TemperatureConverterTest(“testToCelsius”));

    return ts;

    with: var ts : TestSuite = new TestSuite(TemperatureConverterTest);

    doesn’t work.

  8. Theo Says:

    It should work, the code in the article is exactly what I do, save for the actual names of methods and classes. Could it be something else?

  9. Steve Says:

    I able to run the original code correctly. Assuming you didn’t intend to remove the original ‘return ts’ (which invokes an Arg not returned compiler error.

    Running the code produces:

    ArgumentError: Error #1063: Argument count mismatch on CommandStackTest(). Expected 1, got 0. at flexunit.framework::TestSuite() at CommandStackTest$/suite() at ProximityTestRunner/private::createSuite() at ProximityTestRunner/private::onCreationComplete() at ProximityTestRunner/___ProximityTestRunner_Application1_creationComplete() at flash.events::EventDispatcher/flash.events:EventDispatcher::dispatchEventFunction() at flash.events::EventDispatcher/dispatchEvent() at mx.core::UIComponent/dispatchEvent() at mx.core::UIComponent/set initialized() at mx.managers::LayoutManager/private::doPhasedInstantiation() at Function/http://adobe.com/AS3/2006/builtin::apply() at mx.core::UIComponent/private::callLaterDispatcher2() at mx.core::UIComponent/private::callLaterDispatcher() at [enterFrameEvent]

  10. Theo Says:

    Hm, I might just have understood what it was you didn’t get. What I mean by the first part of the article is that you shouldn’t have a static method called “suite” on your test case class. Instead you should create a suite using the TestSuite constructor, passing the test case class as argument.

    So instead of:

    var suite : TestSuite = TemperatureConverterTest.suite();
    

    you should not have a suite method at all, and instead do this:

    var suite : TestSuite = new TestSuite(TemperatureConverterTest);
    

    because that will a) save you a couple of lines of code, b) save you from duplicating the test method names and c) having code that doesn’t rely on a test case having a method which can’t be enforced by the type checker.

  11. Steve Says:

    Much appreciated Theo…it becomes clear it’s not just testing that i’m too noob to. I can make enough sense of the process to see the advantages you are outlining…in particular b) & c).

    But I’m unable to see the specifics of what you are implementing. I don’t see the line: var suite : TestSuite = TemperatureConverterTest.suite();

    in the original. Perhaps i’m getting screwed up with which file is being addressed – the .mxml or the .as

    Or might the fact that i’m coding in beta 3/moxie have anything to do with it?

  12. Theo Says:

    Sorry, I quoted the code from the top of my head. This is how Darron Schall suggested that you set up your testing (for complete code listing see his post, the link is at the top of my article):

    // After everything is built, configure the test
    // runner to use the appropriate test suite and
    // kick off the unit tests
    private function onCreationComplete( ) {
        testRunner.test = createSuite();
        testRunner.startTest();
    }
    
    // Creates the test suite to run
    private function createSuite( ) : TestSuite {
        var ts : TestSuite = new TestSuite();
    
        // TODO: Add more tests here to test more classes
        // by calling addTest as often as necessary
        ts.addTest(TemperatureConverterTest.suite());
    
        return ts;
    }
    

    The createSuite method could instead be written like this:

    private function createSuite( ) : TestSuite {
        var ts : TestSuite = new TestSuite();
    
        ts.addTest(new TestSuite(TemperatureConverterTest));
    
        return ts;
    }
    

    It may not look as much of an improvement, but it proves that the static “suite” method on the test case class is not needed.

    To expand a little on the reasons why I like the second version better:

    a) Less code, not in the “createSuite” method above, but by removing the “suite” method on the test case class. This is the least important actually, and had it not been for reason b) I wouldn’t mind that much.

    b) Code duplication, essentially you write the names of the tests (the test methods) twice, and you have have to manually ensure that the names are repeated. You will forget to keep these in sync more than once during the life of the test case class. Every time you add, remove or rename a test you will have to check the “suite” method to make sure that the names match. An illustration:

    class TestExample extends TestCase {
    
        public function testOneThing( ) : void {
            // ...
        }
    
        public function testAnotherThing( ) : void {
            // ...
        }
    
        public function testYetAnotherThing( ) : void {
            // ...
        }
    
        public static function suite( ) : TestSuite {
            var s : TestSuite = new TestSuite();
    
            s.addTest(new TestExample("testOneThing"));
            s.addTest(new TestExample("testAnotherThing"));
    
            return s;
        }
    
    }
    

    There is no way, except for writing a custom parser, to make sure that the names of the test methods are reflected in the “suite” method. In my example I forgot to add the test “testYetAnotherThing”. I would get no warning from the compiler, nor would I get a runtime error. The tests would pass and I would have no idea that not all the tests had run.

    c) Static methods can’t be enforced. This means that you end up with a regime where your test case classes need to inherit from TestCase, a formal requirement, but they also need a method called “suite” which can’t be enforced by inheritance or any other method but duck typing. It’s really bad OO. Unlike b) this problem will be picked up by the compiler, however.

  13. 1somniac Says:

    I changed var suite : TestSuite = TemperatureConverterTest.suite();

    to : var suite : TestSuite = new TestSuite(TemperatureConverterTest);

    And i have Steve error problem : ArgumentError: Error #1063: Argument count mismatch on CommandStackTest(). Expected 1, got 0.

    but the only file I modified was the main mxml file, maybe there is other change to do ?

  14. Theo Says:

    Sorry, without more information about how the rest of your setup looks I can’t say what the problem is.

    The argument error you get might indicate that you have no constructor on your test case subclass (which is needed in order to pass the name of the name of the test to the super class, another ugly thing about FlexUnit), but in that case I can’t see how you implemented suite, so it shouldn’t be that.

  15. 1somniac Says:

    Thank you for your interest. Here is my the main.mxml file in which i have made the change you said :

    main.mxml:

    <!-- flexunit provides a very handy default test runner GUI -->
    

    The other file are the same as indicated in the tutorial, i’ve changed nothing :

    TemperatureConverterTest class : package {

    import flexunit.framework.TestCase;
    import flexunit.framework.TestSuite;
    
    public class TemperatureConverterTest extends TestCase {
    
        public function TemperatureConverterTest( methodName:String ) {
            super( methodName );
           }
    
        public static function suite():TestSuite {
            var ts:TestSuite = new TestSuite();
    
            ts.addTest( new TemperatureConverterTest( "testToFahrenheit" ) );
            ts.addTest( new TemperatureConverterTest( "testToCelsius" ) );
            return ts;
        }
    
        /**
         * Ensures the celsius to fahrenheit conversion works as expected.
         */
        public function testToFahrenheit():void {
            // Test boiling point
            var celsius:Number = 100;
            var fahrenheit:Number = TemperatureConverter.toFahrenheit( celsius );
            assertTrue( "Expecting 212 fahrenheit", fahrenheit == 212 );
    
            // Test freezing point
            celsius= 0;
            fahrenheit = TemperatureConverter.toFahrenheit( celsius );
            assertTrue( "Expecting 32 fahrenheit", fahrenheit == 32 );
        }
    
        /**
         * Ensures the fahrenheit to celsius conversion works as expected.
         */
        public function testToCelsius():void {
            // Test boiling point
            var fahrenheit:Number = 212;
            var celsius:Number = TemperatureConverter.toCelsius( fahrenheit );
            assertTrue( "Expecting 100 celsius", celsius == 100 );
    
            // Test freezing point
            fahrenheit = 32;
            celsius = TemperatureConverter.toCelsius( fahrenheit );
            assertTrue( "Expecting 0 celsius", celsius == 0 );
        }
    
    }
    

    }

    And the TemperatureConverter class:

    package {

    public class TemperatureConverter {
    
        public static function toFahrenheit( celsius:Number ):Number {
            return ( 9 / 5 ) * celsius + 32;
        }
    
        public static function toCelsius( fahrenheit:Number ):Number {
            return ( 5 / 9 ) * ( fahrenheit - 32 );
        }
    }
    

    }

    I might have forget something…

  16. Charles Dale Says:

    Hoorah, I’m very sick of adding these addTest calls every time I write a new test. No longer!

    Just a note about your suggested rewrite of the isTestMethod function. Surely calling the match function of the string rather than the test function of the RegExp would be easier to read? I.e.:

    private function isTestMethod( name:String ) : Boolean { return name.match(/^test/); }

  17. Theo Says:

    String.match returns an array of matches, not a boolean like RegExp.test. You could use String.search too, which returns the index where the pattern matched.

    But I don’t think it’s very important, you are right in that one should use the method which is most readable.

    return /^test/.test(name);

    return name.search(/^test/) != -1;

    return name.match(/^test/) != null;

    They are all equivalent, but in this case, since the we’re matching against /^test/, think that my version is the worst. Had it been another pattern, say /^\w+/ I would have said it was a draw.

  18. Charles Dale Says:

    Ack you’re right. I’ve grown accustomed to using match() within if statements, which casts to Boolean for free – I forget you don’t get that with a return.

    It’d be nice to have a simple RegExp match boolean function on strings ala RegExp.test(). indexOf gets used for this purpose all the time in code I see, which bugs me…

    That or I’ve spent too much time in Perl and Ruby =P

  19. Stephan Bezoen Says:

    @1somniac: when passing the class TemperatureConverterTest directly in the constructor of TestSuite, function names are no longer passed to the constructor. Therefore the constructor of TemperatureConverterTest is called without parameters. So it works fine if you change the constructor to: public function TemperatureConverterTest( methodName:String = null) Mark that the methodName parameter is now optional, and has a default value of null.

  20. Kelvin Luck Says:

    I started out using FlexUnit as you suggest but I believe I have just come across a use-case for the other approach.

    Basically I have once complex method that I need to test with a large variety of input values. Rather than creating a different test for each possible input value (a large but defined set) I instead loop over all of the possible input values and create a TestCase for each – passing the values into the constructor. The TestCase subclass only contains one test method which uses all of the values passed into the constructor.

    This allows me to have all of the tests run as individual tests so that I can see which one (if any fails).

    I’m interested to know if you think this is a valid use-case (if I explained it clearly enough).

    Cheers :)

  21. Theo Says:

    If I need to know exactly which input value made an assert fail I usually use the message parameter (the first parameter to the assertX methods). Like this:

    (Let y be a dictionary mapping parameter value to expected result)

    for ( var x in y ) { var expected = y[x]; var result = methodToTest(x);

    assertEquals(“Test with parameter value ‘” + x + “‘ failed”, expected, result); }

    If that fails the message will read something like “Test XXX failed: Test with parameter value ‘X’ failed, expected <A> but was <B>”.

    Tests should be self-contained, you shouldn’t use the code that starts the tests to determine which values are tested.

  22. Kelvin Luck Says:

    I can see your point from a theoretical point of view but unfortunately I don’t think your approach would work in my case…

    It turns out that my defined set of input values ends up in 192000 tests once I’d written a function to get me all the possible combinations… Running an individual test for each combination allows me to see progress while the test runs and doesn’t lock up the flash player as each test is run on a different frame…

    I’m not sure what the recommended way of testing a situation like this would be (a complex function which can have 192000 legal combinations of arguments passed in)?

  23. Theo Says:

    Testing all 192000 possible input values seems like overdoing it.

  24. FlexUnit - Some useful examples covering unit testing in Flex and Actionscript 3.0 at building blocks Says:

    [...] Hultberg has some valid critiques of the framework, as well as an alternative approach to the ‘Temperature Conversion’ example that is supplied with the FlexUnit distribution. I found [...]

  25. Marty Pitt Says:

    Hi

    I just found this post, and didn’t realize that passing a calss through to the test suite would discover all the test classes.

    I also am not a fan of the testSuite.add(xxx) method — I went through the pain of this approach for an entire project, and ditched.

    As an alternative, I wrote a custom parser using metadata to allow me to decorate methods with [Test] to generate a test. I still prefer it over the name method approach. (Though probably would’ve have developed it had I known).

    Details on my blog here: http://martypitt.wordpress.com/2008/03/16/custom-metatags-and-slicker-unit-tests/

    Marty

  26. Yevgeny Dorogaykin Says:

    Hi I want to verify some issue which is bothering me very much. As i already have checked if i have for example class with 3 tests: public class TestClass extends TestCase { … public function test1():void { }

    public function test2():void { }

    public function test3():void { } } we’ll always run(means creates instances) this class 3 times for each test? And it doesn’t matter if i using function suite( )approach or dynamic look-up approach.

    Is it true?

    I’m asking it because i have class with very long and heavy start-up. And when i testing it i want to perform this init only once. There is no problem with this in NUnit for example. But i can’t figure out how to do it with flexunit.

    Thanks in advance.

  27. Theo Says:

    Yevgeny,

    I’m not sure I understand exactly what you ask, but this is what would happen with your example test case:

    1. an instance of TestClass is created
    2. setUp runs
    3. test1 runs
    4. tearDown runs
    5. another instance of TestClass is created
    6. setUp runs
    7. test2 runs
    8. tearDown runs
    9. a third instance of TestClass is created
    10. setUp runs
    11. test3 runs 12 tearDown runs

    Each test is run in its own instance of the test case class.

    If this bothers you very much switch to dpUnit which does this instead:

    1. an instance of TestClass is created
    2. setUp runs
    3. test1 runs
    4. tearDown runs
    5. setUp runs
    6. test2 runs
    7. tearDown runs
    8. setUp runs
    9. test3 runs
    10. tearDown runs

    …which makes more sense. dpUnit also supports asynchronous setUp and some other things that FlexUnit don’t have. I’ve switched myself and I’m not looking back.

  28. Yevgeny Dorogaykin Says:

    Thanks Theo for your answer. You’ve understood my question right.Actually I prefer 3 possibility: 1. an instance of TestClass is created 2. setUp runs 3. test1 runs 4. test2 runs 5. test3 runs 6. tearDown runs

    That’s what I know from Nunit. For me it’s more natural behaviour. I definitely will check dpUnit.

  29. (B)logger vom hogger » Blog Archive » Testdriven development (TDD) with Adobe Flex, Air Says:

    [...] bad examples, poor documentation, some bugs related to asynchrous testing. Some issues are already discussed in this blog entry. If you want a “good” starting you should read [...]

  30. cholm Says:

    One case which I believe you may overlook in which running tests in a given order makes sense is when running integration tests against some stateful entity, such as a web services layer that interacts with a database.

    You express rightly, and very well, that unit tests should have no external dependencies and should manage their own state, however unit tests are not the only type of tests written with this, and other, frameworks. Very nice article, thanks from the community at large for your efforts.

  31. Theo Says:

    Well, I don’t agree that integration tests should work any different. They too need to run in isolation. Perhaps not isolation from the database, if that’s what you’re testing, but definitely in isolation from each other. If they don’t you have no way of knowing which test actually failed.

    If one test depends on another you have a fragile test suite. Refactor and introduce a setup that puts the system in a state suitable for both tests (or break them apart and have separate setups). If tests absolutely must depend on each other then they should be explicitly written in sequence, relying on the framework running things in alphabetical order seems like inviting trouble. It also makes it less obvious what the tests do and what they actually test. One reason for writing tests (although not the main one) is that they describe how the system works. Depending on tests running in a particular order (which isn’t necessarily obvious) makes the tests less understandable.

  32. Patrick Says:

    Great article. Unfortunately, even if FlexUnit manages to fix all it’s problems in the next release, Flex itself still throws some serious roadblocks in the way of good unit testing. For example here is a quote from the flex 3 livedocs:

    “Variables representing objects, arrays, and functions are compared by reference. Two such variables are equal if they refer to the same object, array, or function. Two separate arrays are never considered equal, even if they have the same number of elements.”

    What that boils down to is that there is no way to do equality tests on Objects or Arrays without writing your own custom equality functions. I’ve tried writing a simple test that compares the data sent from a form to the data that comes back after some server side processing and this limitation has made it pretty much pointless. I would need unit tests for my unit tests and that just doesn’t make sense.

  33. 1ndivisible Says:

    Hi Theo,

    I think I have a good use-case for ordered tests. I am writing a class that uses SWFAddress. SWFAddress uses all static methods and its SWFAddress.as class is reliant on communication with its js counterpart in the browser. The first time you register an event listener for its CHANGE event, it fires off its INIT event. I need to test interaction with the class and therefore SWFAddress before the INIT event is fired and after it.

    There is absolutely no way to tearDown SWFAddress, and without any way to control the order of the tests, I can’t ensure that one group of tests occur before INIT and one group of tests occur after INIT. In this case, I think the only solution would be control of order.

  34. Theo Says:

    @1ndivisible you are writing an integration test using an unit testing framework, so the fit will not be perfect. the problems you’re having are more down to you not being able to control the situation and run the tests in isolation. To me it sounds more like you’re actually running one test, even though you’ve split it up into different methods. It’s more or less like having multiple asserts in the same test method, one has to run before the other. The difference is that in your case the tests after a tests that fails still run — but since one failed you can’t trust the result of the others anyway, so in practice there’s no difference.

  35. 1ndivisible Says:

    This is true. I never considered it as integration testing, but I suppose that is exactly what it is.

  36. Paul Williams » Unit Testing User Interfaces - Introduction Says:

    [...] The lost FlexUnit documentation [...]

Leave a Reply