Sunday, March 23, 2008

Database Dependent JUnit Tests and the “Dependent Object Framework"

10/27/2008 --> Please stay tuned for an update to this article in the next couple of days.

Have you ever worked on a large enterprise software project (one that heavily uses a database) and wanted to add JUnit tests? And did you quickly realize that it’s easier said than done? Why is that? Want to learn a better way? If so, keep reading how a new open source project can solve this problem.

Back in 2004, I managed to introduce a huge body of JUnit tests in my team’s project, IBM’s “WebSphere Product Center,” but the tests solely focused on the core storage layer which had few dependencies to the rest of the complicated system. Managers kept pestering for more JUnit coverage. The conventional wisdom was that we needed to refactor the code into the appropriate layers so that dependencies on the database could be isolated and “mocked out”. The conventional wisdom (and most definitely the managerial wisdom!) was that thou shall not make big code changes without lots of automated unit tests. Hmmm, this sounded eerily like “Catch 22”. We couldn’t change the code without JUnit tests and we couldn’t have JUnit tests without changing the code!

We resigned ourselves to the reality that we had to have JUnit tests that ran against the database. What were our options for setting up the database to run tests?


Technique

Details

Issues

SQL Scripts

Write inserts to populate a blank database.

Any time the database schema changes, the scripts need modification.

Restore a backup of the database

Use the UI or any means to get the database ready for the unit tests. Then save a backup copy.

Any time the database schema changes, the backup copies need to be recreated.

Write Java code

Write Java code (probably many lines) to setup the objects for the test.

Too much Java code to maintain, and bugs with the copious code.



All of these options were painful, fragile, monolithic, and tedious. If you’ve tried any of these options, then you would agree.

A co-worker and lisp aficionado, Umair Akeel, came up with a proposal that at the beginning of a test, we list the database objects that must exist for the test. Let’s suppose we are writing a test on an Invoice object and thus that test would require a Customer object, a couple Product objects, and those product objects would require a couple Manufacturer objects. What would that look like?

We would have a method called “require” that would take a file name that would specify a file that would describe a given object, and the file name would also specify the object’s type and primary key. In this example, we would list the following at the beginning of the JUnit test on an Invoice:

require(“customer.25.xml”);

require(“manufacturer.12.xml”);

require(“manufacturer.30.xml”);

require(“product.13.xml”);

require(“product.31.xml”);

In this example, Product 13 depends on Manufacturer 12 and Product 31 depends on Manufacturer 30. The strings passed as parameters to the “require” method specify the object type and the object’s primary key, as well as a file that uniquely defines the object.

I later made the suggestion that instead of having to remember to list out all dependencies (in this case, listing “manufacturer.12.xml” before “product.13.xml”), we put the dependencies of each object in its own object description file.

Then we would only need this at the beginning of the test (or test fixture):

require(“customer.25.xml”);

require(“product.13.xml”);

require(“product.31.xml”);

The “require” method is smart in that it only creates the object in the database if the object does not exist in the database, and once the object is fetched from the database, the framework keeps the database object cached in a map.

This framework allowed us to be very modular in setting up objects for running a database test. That allowed us to achieve the following necessities of running JUnit tests effectively:

  1. Tests must run in any order, and any number of times, with the same result.
  2. Easy to add a new test, and run it without worrying about the state of the database.

We’ve got scripts to set up a blank database, ready for our customers. However, we use the “require” framework to check that the required objects for a test exist. This framework has allowed us to achieve far more legacy test coverage than we ever would have otherwise.

In 2007, I got IBM’s permission to convert the technique into an open source project, which I’ve named the “Dependent Object Framework”.

The Dependent Object Framework (DOF) (http://sourceforge.net/projects/dof/) enables efficient JUnit testing and Test Driven Development against code that depends on objects that are persisted (e.g., database).

Let’s look at a simple example using the Dependent Object Framework. Consider the invoice example listed above. Your invoice needs a product record in the database. So you simply put this line at the top of your JUnit test:

Product product = (Product) DOF.require("product.13.xml");

What you have not done is pre-load the database with a SQL script or database backup. Your test is just saying “make sure that the product record 13 exists in the database and give me that object. You know that the product record will have other records that it depends on, but you only need to specify that your test requires the product record. In fact, maybe the file “product.13.xml” is used by another test, so you didn’t even need to create it.

Can it get any easier than this for specifying what your test needs on top of an unpopulated database?

The DOF makes it easy to write tests with database dependencies. Advantages include:

1. Facilitation of testing legacy enterprise code lacking JUnit tests. With the DOF, you do not need to untangle the database dependencies in order to write mock objects.

2. Easy reuse of database objects needed for unit tests. This facilitates team leverage in creating JUnit tests. The work in creating the database setup for JUnit tests is distributed and modular, versus the monolithic approach of using a SQL script or database backup.

3. Very simple to define which objects a test depends upon. Any indirect dependencies (dependencies of the dependencies) are specified in the files defining the dependent objects. Thus, a JUnit test can depend on object A, and object A might depend on object C. The test simply needs to specify object A, and the definition file for object A will specify object C.

4. Very easy to delete persistent objects created for a test. When one is tweaking tests, this is very helpful.

5. Very easy to add support for new database object types.

6. Fabulous for creating “integration” JUnit tests that run against the database and other real dependencies, just as customers will use the product. So even if you have mocked out the database for many tests, you can still leverage the DOF for running integration tests.

So how does this work?

1. You define a data file format for your object type, such as XML.

2. You write a “handler” class for this data file format. The handler class implements an interface DependentObjectHandler composed of “create(objectFileInfo)”, “get(objectFileInfo)”, and “delete(objectFileInfo).” ObjectFileInfo is a struct with 4 fields: file path that describes the object (i.e., "product.13.xml"), the object's primary key (i.e., 13), the object type (i.e., product), and the file type (i.e., xml)

3. You specify the mapping of the object types to the handlers, in a file called handler_mappings.properties. For example, to specify that files named like product.13.xml map to the ProductXmlFactory class:

product.xml=myPackage.ProductXmlFactory

4. You create a data file for the object you want to create, named like {objectType}.{primaryKey}.{fileExtension} such as “product.13.xml”

5. The data file contains comment lines indicating its dependencies. For example, this line indicates that the product 13 depends on the manufacturer 33 existing in the database.

You can download the framework including the JavaDoc and an example that uses simple XML and hsqldb here: http://sourceforge.net/projects/dof/

Please let me know if you find this useful or if you’d like to contribute. I’m working on extending the example to support hibernate JPA. I suspect that the project might not need to change for this support.

I’m also looking for volunteers to port this code to:

  • C++
  • C#
  • Groovy
  • Other xUnit supported languages.

I'd like to hear if you have any comments or questions on this project.

Cheers,

Justin Gordon

justingordon at yahoo.com

13 comments:

Unknown said...

Really very great information for that post, am amazed and then more new information are get after refer that post. I like that post.
Java Training in Chennai | Java Training Institute in Chennai

Praylin S said...

Very adorable post with creative writing. This is really worth reading. I'm glad that I came across your article. Keep us updated.
C C++ Training in Chennai
C++ Training in Chennai
Unix Training in Chennai
Unix Shell Scripting Training in Chennai
LINUX Training in Chennai
LINUX Course in Chennai
C C++ Training in T Nagar
C C++ Training in Anna Nagar

janathan said...

technical improvement.....
Free Inplant Training Course For ECE Students
INTERNSHIP
INTERNSHIP FOR AERONAUTICAL ENGINERING STUDENTS IN INDIA
INTERNSHIP FOR CSE 3RD YEAR STUDENTS
Free Inplant Training Course for Mechanical Students
INTERNSHIP FOR ECE STUDENTS
INPLANT TRAINING FOR CIVIL
INTERNSHIP AT BSNL
INTERNSHIP FOR 2ND YEAR ECE STUDENTS
INTERNSHIP FOR AERONAUTICAL STUDENTS

janathan said...

great....
COMPANY INTERVIEW QUESTIONS AND ANSWERS
HACK FLIPKART WALLET
DATA STRUCTURE
TYPE SCRIPT GETTING ERROR
APTITUDE NUMBERS
APACHE PIG COUNT FUNCTION
APTITUDE PROFIT AND LOSS
SUBMIT IS NOT A FUNCTION ERROR IN JAVASCRIPT
THE CP OF 15 BOOK IS EQUAL TO THE SP
WHAT NUMBER IS TO BE SUBTRACTED

Vijaykumar said...

very nice blog..
Inplant Training in Chennai
Iot Internship
Internship in Chennai for CSE
Internship in Chennai
Python Internship in Chennai
Implant Training in Chennai
Android Training in Chennai
R Programming Training in Chennai
Python Internship
Internship in chennai for EEE

Vijaykumar said...

tecnical knowlege improvement blogs..
Crome://Flags
Python Programming Questions and Answers PDF
Qdxm Sfyn Uioz
How To Hack Whatsapp Account Ethical Hacking
Power Bi Resume
Whatsapp Unblock Software
Tp Link Password Hack
The Simple Interest Earned On a Certain Amount Is Double
A Certain Sum Amounts To RS. 7000 in 2 years and to RS. 8000 in 3 Years. Find The Sum.
Zensoft Aptitude Questions

Faizal said...

Excellent works!!!Information's are amazing visit here for more...
Java training in chennai | Java training in annanagar | Java training in omr | Java training in porur | Java training in tambaram | Java training in velachery

jenani said...

Great information. Finally, thanking the blogger to launch more further too.
Java Training in Chennai

Java Training in Velachery

Java Training in Tambaram

Java Training in Porur

Java Training in Omr

Java Training in Annanagar

vanathi said...

very nice blog and thanks for sharing this very amazing content with us ..keep post this type of blog ..we support your all blog .
Software Testing Training in Chennai

Software Testing Training in Velachery

Software Testing Training in Tambaram

Software Testing Training in Porur

Software Testing Training in Omr
Software Testing Training in Annanagar

subathara said...

Really very great information for that post, am amazed and then more new information are get after refer that post.Digital Marketing Training in Chennai

Digital Marketing Training in Velachery

Digital Marketing Training in Tambaram

Digital Marketing Training in Porur

Digital Marketing Training in Omr

Digital MarketingTraining in Annanagar

Tiger Online shop said...


You can get Apple-certified repairs and service at the Apple Store or with one of our Apple Authorized Service Providers.
mobile phone repair in North Olmsted
Worked as a Senior SEO & Digital & Social Media & Graphics Design & cpa & Drop shipping & Video Editing And Youtube & Web Design And Development & Affiliate Marketing trainer at BITM (BASIS Institute of Technology & Management) since 2014-2018. Successfully completed 50+ SEO batches, 20+
Affiliate Marketing batches and 30+ workshop on Freelancing under SEIP (Skills for Employment Investment Program).
outsourcing training courses in uttara

trublogger said...

There is a unit of numerous options that this application has. We suggest you use the newest version of Tally ERP nine unharness. Tally ERP 9 9.6.7 Crack

Anonymous said...

An quick bonus are bonus funds which are be} awarded in a restricted state and additionally be} wagered with immediately upon grant. In general, making a “pass line” bet on whether or not or not the shooter will win provides you 50/50 odds. Blackjack is mostly about luck, and generally, the home edge , 카지노 is just about 2%. Fully adaptable assembly areas, unparalleled leisure and exciting facilities.