Thoughts on Behavioral Testing

I'm kind of amazed with the fact that we define functions or methods as the "behavior" that an object exhibits, but when we test we only care about the outcome of that behavior.

If we have a car object with a driveHome() behavior, we only care that when the function is over we've asserted ourselves into the testing easy chair. Nevermind the fact that on the driveHome() we also errand.stopAtMarket(), errand.mailBills() and friends.visit( "Bob" ), which took too much time and delivered us home too late for our favorite show.

Behavioral mock objects are just one way to be sure that is DOING what you expect it to do, not just RETURNING what you expect.

Matchers

I just ran into something that brought my unit testing to a halt. Strings. Not just any strings, but UUIDs in this case. The code under test calls a mutator method with a generated UUID, like so:

<cfset myObj.setId( createUUID() ) />

How do you set up that behavior? Well, in the current version of CFEasyMock, you can't do that. You can monkey with the MockFactory.equalsArgs() method and add an exception (not the error kind) in the logic to check for a specific string and then return a match, but that's a bit of a hack.

Enter matchers. v1.6 will add the following pattern matchers to the MockFactory object:

isA( Component ) Matches if the actual obj is a component of the type given.

matches( regexpr ) Matches if the actual value matches with the regular expression given.

startsWith( String )
contains( String )
endsWith( String ) Matches if the actual value startsWith/contains/endsWith the given value.

anyBoolean()
anyString()
anyNumber()
anyComponent() Matches any value.

The expected usage for the uuid problem could be solved in a number of way, but the matches() would provide the best comparison. Using anyString() would be too loose.

<cfset mf = createObject( "component", "easymock.MockFactory" ) />

<cfset user = mf.createMock( "my.User" ) />

<cfset mf.expect( user.setId( mf.matches( "^[A-Z0-9]{8}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{16}$" ) ) ) />

<cfset mf.replay( user ) />

...

<cfset mf.verify( user ) />

Hmmmm... looks like I might have to add an anyGUID() function as well.

Arguments and Return Values and NULLs ... oh MY!

I had thought the argument monster had finally died with my last post. But noooo. There's yet another wrinkle, and this time it is actually not my fault.

Consider the following code.

<cffunction name="foo">
<cfargument name="param1" required="false" />
<cfargument name="param2" required="false" />

<cfdump var="#structKeyExists( arguments, "param1" )#" />
<cfdump var="#structKeyList( arguments )#" />
<cfdump var="#arguments#" />

<cfset bar( argumentCollection=arguments ) />
</cffunction>

<cffunction name="bar">
<!--- do nothing --->
</cffunction>

<!--- Run the function --->
<cfset foo() />

Okay, so guess what you see? First, you'll notice the obvious "NO" coming from the structKeyExists() function. We expect that, and love it for behaving as expected.

And now it just starts getting a little weird. You'll notice the list of arguments has both "param1" and "param2" in the list. What? I didn't pass in any values why would they be in the list, especially if they don't 'exist'? Well, take a look at the next output.

The dump of the arguments struct provides the answer. Notice that pesky little phrase 'undefined struct element'? CF has predefined the argument collection keys, and then places the corresponding values. Since there was no value passed, the java side of CF places NULLs into their slots. The structKeyExists() function correctly determines existence if the value is NULL, but that's in java. CF doesn't have a way to directly detect for nulls.

Be very careful when dealing with non-required parameters and then passing the arguments struct off as an argumentCollection.

Just for kicks try this:

<cffunction name="nullify">
<!--- do nothing --->
</cffunction>

<cfset foo = "Hello World!" />
<cfset foo = nullify() />
<cfoutput>
#foo#
</cfoutput>

It's easy to spot the problem here, but in real world code where the offending function is more than likely in another file, it can cause a bit of a headache.

Be default the java methods that the CF code gets translated into will return a null value if a return value is not specified. The nullify function above essentially is the same as:

<cffunction name="nullify">
<cfreturn javaCast( "null", "" ) />
</cffunction>

CFEasyMock v1.2 update

First, I've updated the CFEasyMock download zip to include CF7 again. Unfortunately where I work, the decision to upgrade to CF8 has been held off. Mostly by management, isn't it always?

So much of the work I've actually been able to devote to the package has occured on the CF7 version. Luckily, the primary difference between CF7 and CF8 versions of CFEasyMock lie outside of CFEasyMock itself, in the reflect package. The changes to CF7 were easy to integrate back up to the CF8 version.

Versions prior to 1.2 used a component called ExpectedMethodCall to hold the called method (as a reflect.Method reference) and the argument parameters for that call. Functions of ExpectedMethod call would do the matching to see if the expected and actual calls matched. Since this code is obviously the same for all method calls, it was moved into MockFactory, eliminating the need to create extra components just to perform matching calculations. The changes are completely internal and no unit test code needs to be altered.

Performance Enhancements

The reflect package which CFEasyMock relies on to actually create the proxy has been updated to support the CF8 feature of onMissingMethod.

Nothing in the easymock package has changed. However, I will be updating the online zip and the SVN repository with the CF8 version of the Proxy component.

You will probably see a new project come up in a few days that will contain the reflect package itself. This is the package that does all the heavy lifting to create the proxy components and there are some cool ideas I'd like to implement, and these things aren't directly tied to the easymock package.

It will also allow me to version them seperately.

The primary benefit of moving to CF8 and onMissingMethod is speed. The CF7 version of reflect has to write a dummy component with the correct function call, create that file as a component instance, and then inject the function into the proxy. This can take considerable time if you have many mock objects.

Below are some times from unit tests for the ExpectedMethodCall, one of the biggest test cases in the package:

TEST CASECF7CF8
------------------------------------------------------
testEqualsMethodCallValid2140ms1156ms
testEqualsArgsFailStructCount1203ms625ms
testEqualsArgsFailComponent1218ms625ms
testEqualsArgsFailSimple1172ms641ms
testEqualsArgsFailQueryColumnlist1188ms609ms
testEqualsMethodCallInvalid1469ms890ms
testEqualsArgsFailArgCount1265ms625ms
testEqualsArgsFailQueryRecordCount1203ms610ms
testEqualsArgsFailStructKey1219ms625ms
testEqualsArgsValid1907ms609ms

Money where my mouth is...

You would think that given the context of the CFEasyMock package I would have a slurry of unit tests covering all aspects of the project from a very TDD perspective.

Alas, that is not the case. There were scattered tests that didn't cover much, other than basic creation of mocks and simple happy path tests.

Well, that has changed, or at least has mostly changed. I've gotten about 40% coverage now, and the unit tests use the CFEasyMock package.

I've found that the greatest benefit of CFEasyMock so far, has been the near elimination of having to do stepwise debugging to find out what a particular method is doing. You know the drill, code in output so you can see where things are happening. Run the method, dump values, see where things are occuring. Not so much anymore. I just use the strict mock and declare expected behavior.

I've added a tweak to the MockFactory and MockInvocationHandler that allows you to switch a mock into debug mode, collecting all method calls/return values as they go through, regardless of mode or expectation. The generated stack trace can be accessed by the stackTrace() function in the MockInvocationHandler itself.

<cfset mf = createObject( "component", "easymock.MockFactory" ) />
<cfset method = mf.createMock( "reflect.Method" ) />

<cfset test = createObject( "component", "test.Blank" ) />
<cfset other = createObject( "component", "sample.Collaborator" ) />

<cfset mf.expect( method.invokeMethod( test, structNew() ) ).andReturn( "Hello!" ) />
<cfset mf.expect( method.invokeMethod( other, structNew() ) ) />

<cfset mf.replay( method ) />
<cfset mf.debug( true, method ) />

<cfset method.invokeMethod( test, structNew() ) />
<cfset method.invokeMethod( other, structNew() ) />

<cfset mf.verify( method ) />

<cfdump var="#method.getProxyInvoker().stackTrace()#" />

I've also located and fixed the following issues:

  1. expects() was not triggering correctly due to the ResultMapper.add() not handling the first added request correctly.
  2. times() was overwriting the previous Result as well as the Range, instead of just modifying the previous Range.
  3. Component arguments were not compared at the name (aka type) level, which was allowing two of the same method call with different components to pass.
I've uploaded the latest build incorporating these changes v1.0.003

Chugging away...

Ryan Wood and I have been working through a few issues. We've uncovered and fixed a bug with the expect() method. There's a new zip with the latest version on the project site, v1.0.002

Ryan has also proposed a solution for allowing CFEasyMock to work with injected functions. I'll be working on getting that completely integrated this weekend.

Thank you Ryan!

Named vs Unnamed Arguments in CF functions

ColdFusion gives developers a great deal of flexibility when creating functions. Not all arguments have to be known at runtime by declaring them with a tag. Any additional arguments passed get a number key relative to their position (if they aren't passed in by name or via an argumentcollection).

So for function foo...

<cffunction name="foo">
<cfargument name="bar" /.
...
</cffunction>

called by

<cfset foo( 10, 20, 30, 40 ) />

Would yield an argument struct of: bar = 10, 2 = 20, 3 = 30, 4 = 50

If you call foo like the following:

<cfset foo( bar=10, var1=20, var2=30, bar2=40 ) />

you would get an argument struct of: bar = 10, var1 = 20, var2 = 30, bar2 = 40

At first I thought this was a great concept. Less typing, not constrained to a particular implementation, etc. Then I got to thinking. This could really turn into a slippery slope.

Functions that don't have a clear set of arguments can easily end up not having a clear definition of purpose. (Not to mention difficult to debug!) They could start to take on more and more behavior and then lose any resemblance of reusability.

Functions that perform one specific behavior will have a known set of inputs. Those input values may vary, but the inputs themselve are always known.

There are some instances where unnamed arguments does make things a lot easier. There's even an example of that within CFEasyMock itself, in the MockFactory.replay() method (and reset() and verify()). It is simply easier to pass in a series of components rather than make an array out of them first. (Which would be the most prudent way to do it.) However, there is still the chance that someone passes in the wrong object, but that wouldn't be caught until runtime anyway.

It's a double edged sword. There are definite benefits, but it should be used sparingly with good reason, as opposed to a general coding practice.

CFEasyMock finds a home

The company I currently work for is primarily a java shop. The java development side of the house already has a lot of process around TDD and unit testing.

One of the advantages they have in their TDD and unit testing frameworks is a java package called EasyMock. It allows the developer to create tests that can verify behavior of functions. Normally, when we right unit tests we are only asserting some variable state. What actually happens in the function to change that variable state is a mystery.

Behavioural mocks allow for interactions between the function under test and the collaborator component(s) to be checked for expected behavior.

I ported over the EasyMock package for two reasons. 1) The CF development processes are starting to take on more agile development aspects. We need to define function behavior during testing. 2) We also needed mocks that work under CFMX 7.x, as we don't run 8.x yet.

BlogCFC was created by Raymond Camden. This blog is running version 5.5.006. | Protected by Akismet | Blog with WordPress