Summary/Abstract
Basically, instead of a focus on "unit testing", developers should be focused on any "real world test" that will help the software project become better. If these real world tests can be done using unit tests, then so be it - but many times they cannot.
What is Unit Testing?
When I first read about unit testing many years ago, I was flooded with doublespeak and buzzwords explaining why "Unit Testing" was important.
I didn't understand what they were getting at, due to all the funny English they used.
Then it struck me - the reason they were using so much doublespeak was because "unit testing" was actually common sense. You couldn't write a book about common sense - you had to make it sound complicated. You couldn't go into depth about testing if testing was common sense. Although testing is common sense, sometimes developers don't write or perform tests simply because they don't feel like it or they have egos and assume their code is good. That's basically the summary of what all the Unit Test books out there are trying to get at.
Everyone does unit testing, you just didn't know it
The first time a programmer experiences "Unit Testing" is probably during writing his first hello world program, or his first procedure or function, or his first class, or his first computer program. You write a small little function for your first program and launch it.. hoping it will work, but you don't know for sure yet. Then you think.. Gee, maybe I need to test my function or procedure to see if it actually works.. because this is my first time programming.. hmm.. let's start the program and see if it works. Should I test some more values in my program or more parameters or messages, or is this just a throw away program that I won't ever need again, or that I will only use 5 times?
Tada! You've done simple "Unit Testing" on your first day of programming, without even knowing what you just experienced was "Unit Testing".
But did you do enough testing
Taking this further, if your application is more important and you know it has a long term future, you do more tests (real world hopefully, not necessarily unit testing but live real world environment testing), especially on the parts of the program that are really important. In some cases the really important parts of the application may be security, or functions that you just recently wrote which are brand spanking new, or functions you don't know if you coded perfectly since they were not trivial. After all, just because it compiles, or just because the interpreter says the code is okay, doesn't mean from a user or customer perspective that the code will be okay. Many of you are thinking right now reading this article.. gee, testing some code you wrote... that is common sense, no? Well it is. That's what startles me about "unit testing" - is that it is basically common sense.
Testing is common sense, yes. Would you drive a car from a factory if that car had never been tested for safety? However, some programs are not at all like cars - testing a throw away application for example is definitely a waste of time, if the throw away application works for you right now. You don't always test an apple for poison before eating it - because if you tested everything in life you would not have time to live. This is the major problem with Unit Testing - deciding what actually is important to test, and if the environment is real enough for the test to actually be worthwhile.
Do I start writing tests now or later?
I once wrote an application that grabbed over 15,000 text snippets and placed them into a database - I wrote no tests and got the data into the database. Even though it was a lot of data, it seemed to go really smoothly on the first few tries. The client needed a quick glimpse of the data which he may need perfected - depending on what he saw. Since he may need more of the data, I won't spend time writing tests until I know further details.. you don't want to spend 30 hours testing data or software that somebody may need more perfected. In fact, the client ended up needing more data but he said the data was already perfect as is, and he wanted more features implemented since he may use the data based on what the results of the next feature showed (a few more database columns). So if the customer still may need the data and is already satisfied, I won't spend 30 hours writing tests yet, or even 1 hour writing tests - because it isn't definite yet as to whether writing tests is important at this point or not.
Real World Testing vs Unit Testing
The biggest problem I have with "Unit Testing" is that sometimes too much focus is placed on "source code testing", and not enough testing is done with real world user input or real world environments where the settings may differ extremely.
For example if you are building a website and you want to test the security of your URL variables or input form POST/GET variables, the best way to test your web program is to visit your own site, and inject cracker/hacker text into your program. You could even record a bunch of your cracker/hacker exploits into your text editor and save them for later use. You could go further and retest the same exploits on different parts of your site by writing a macro to do it, or using cURL - but be warned that cURL isn't a real world testing environment since your customers will be using a web browser, not cURL. This is real world testing rather than unit testing at this point.
In addition to real world testing (not unit testing yet), you can simply study your code and verify that you have filtered all your incoming variables. You may quickly notice some code that is wrong before you even think about or get to unit testing. Again, common sense in place. You may not notice everything - and real world testing will catch more issues, and unit testing may catch some issues too.
Writing some simple tests might prove that your functions work - but how many tests do you have to write, and are your tests really legitimate real world tests from a real world environment? Remember, unit tests are sometimes launched from non-real world settings, such as a software program that runs through the tests. Your customers aren't software programs - your customers are real people with keyboards, mice, and brains.
Do your unit tests truly emulate the real world environment? Is your environment simple enough that the unit tests will be legitimate?
In some cases, unit tests do emulate the real world environment enough - for example if you are testing a compiler for correctness, the compiler can be sent a source file through a software program, and the keyboard, gui, and mouse have little effect on the test. Unfortunately, in many cases, unit tests do not emulate a real world environment enough. A compiler, parser, or function that has an input and an output value are the easiest pieces of code to test - but unfortunately software isn't this simple.
Customer Testing?
Are you putting yourself in your customers shoes? Do you have a real world customer environment to test in?
Have you put yourself into cracker/hacker's shoes, and do you really know how a cracker/hacker thinks? Remember, the hacker/cracker doesn't exploit your program by writing unit tests, he exploits it in a real world testing environment. Some customers or users are not even crackers/hackers, but they can accidentally input incorrect values that are similar to what hackers attempt at times. Therefore putting yourself into the hacker/cracker's shoes is still useful, even if your program is so private that it will never see a hacker.
Are your tests REALLY needed for feature XYZ?
Are you just going to write some tests that satisfy your function in the simplest form - and are your tests so simple that they may not really emulate the real world environment? Can you narrow down your tests enough so that you don't spend endless years writing more tests after tests? For example, have you tested your program with a Chinese keyboard and a Russian keyboard - and do any Chinese people ever use your program, or is it just two pure English people that will ever use the program? Again some common sense in place - you still have to prioritize what is really worth testing, as testing is an infinite road. If you start writing test that verify a Chinese keyboard works, and no Chinese people ever use your application - you will be wasting time.
I've thought about writing tests for my programs to verify UTF-8 is working correctly, since I assumed the data I would be dealing with would be international data. Sometimes UTF-8 encoding can go wrong and needs to be tested. But after studying my customers/users needs, they may tell me that he/she isn't even interested in the international content and only deals with USA data and he will never need UTF-8. So before thinking about tests and before inventing tests, you still have to prioritize and find out what to test. Is this up front design and up front thinking? I think it is up front questioning.
Question your customer before writing tests and before doing real world tests. Is talking to your customer ahead of time up front design? A bit - you aren't diving into the code - you are doing planning - so yes it is up front thinking. I know all the extreme programmers are against up front planning - but hey, you have to talk to your customer, and talking to your customer is up front. Don't write tests for things you won't need to test - a lot of testers write tests that test features which will never be needed, because they didn't do enough up front discussion with their client - they just dove into code.
Another problem that can occur with unit testing or real world testing, is testing features which the customer stated as important but were actually not. Sometimes the customer changes his needs. If you did a whole bunch of testing (days) to get a certain column of the database working near perfect, and this is the toughest column in the database to test - you had better verify with your customer that this column is actually this important to him. He may tell you that the other 50 columns are more important and this column is now just optional and not needed. You may have spent days testing a feature which you thought was important, but in fact wasn't.
Again, basically common sense - don't test just because someone says you should do testing. Use common sense and find out what it is that you should prioritize to test first.
Real world example: testing security
In the case of web security, logging on to your website and shoving all sorts of different cracker/hacker like pieces of text into your website is probably ten times more important that unit testing, since this emulates a real world environment. Sometimes "Unit Tests" don't emulate a real world environment, and since a real world environment is not emulated it means that your tests may slip by successful even though in the real world the situation isn't the same. For example, did you remember that in the real world, sending in a NULL to a url variable requires that you first send a %00 to the web browser and not an actual null character from the clipboard? Does your test emulate your real world environment - and how long will it take you to reinvent your real world environment with a "Unit Test" framework? Is it faster to just do actual Cracker/Hacker testing, by placing yourself into a hacker/cracker's shoes and visiting the site? Is it possible that a real world environment might have settings that were unanticipated in your coded unit tests? Is it possible that beta testing and cracking is more useful than unit tests in many security situations? Yes.
In this article I mainly focus on a real world testing example: security. However my comments still apply to any sort of testing. Many times real world testing needs to be done instead of focusing too much on reinventing the real world with code testing, i.e. writing tests by inventing the tests yourself. The tests you invent yourself may have no bearing on what happens in the real world with all sorts of variations and gotchyas that can pop up, such as the way browsers handle things differently than a simple function that you wrote does. Does your unit testing, for example, take into consideration 301 and 302 browser redirects, and does it take into consideration a server that may have more users in the real world environment (1000 at a time, say)? Does your unit test just emulate 1 user and not 1000 users? How are you going to emulate 1000 users with code? If you could organize 100 of your friends to test your system out, and have them hammer your server like it was a nail, then it will be a better real world test than a unit test.
Will it take you much more time to emulate and reinvent this situation, instead of just utilizing your real world tools that you have on hand.. such as your web browser and your elite hacking skills? Can you possibly write a test for every single situation that could go wrong or right? Is there a high priority of one test over the other, such as functions that are more prone to pissing a customer off, or functions that are more prone to security leaks?
You have to know what to test first
Some unit testing, believe it or not, requires a short up front plan. I know all the extreme programmers are against up front planning - but you need to prioritize what to test. Testing is an infinite thing - it isn't definite. You can never ever finish testing. So you need to prioritize what needs to be rigorously tested. Sometimes figuring out what needs to be rigorously tested is as simple as putting yourself in the customers shoes and acting as if you were him. You can't just start writing unit tests like mad - you will never finish - tests are infinite. Remember that. You have to decide what to test, and how to test it so that it is like the real world.
If you are just testing an IntToString function, it is as simple as creating a list of all the possible integers, and ramming them through the function one by one (in a loop or otherwise) and verifying that they do convert to a string. You could also test for larger integers that shouldn't fit, or negative numbers, etc. But this is a very very simple function to test - and most of the time in the real world, tests require much more of a hacking/cracking attitude. After all, your testing attitude should be either: "I want to break this program, I am a hacker" or "I want to use this program, I am a customer who puts all sorts of funny inputs in the edit box and I don't pay attention to any rules or read documentation".
Test everything?
There are cases where "testing" is a complete waste of time. For example sometimes when I'm whipping up a quick Regex in a text editor for a find and replace, I don't even care if the regex doesn't work.. because I will just undo my changes with CTRL-Z and repair the regex on a trial and error basis. IN this case, writing unit tests would be a complete waste of time since you are looking for a really quick situation, and you can just test the regex in the real world environment right there and then.
If your software has a deadline and your project already works extremely well without any unit testing, it may be better to just ship the application early and impress the customer - writing unit tests may get your project canceled if you are too late. On the other hand, if you have spare time to write some extra tests (usually projects are accepted late in the software industry) by all means do real world tests and unit tests. I think more importantly are the real world tests over the unit tests though, since ultimately the software will be entering the real world at some point. In the case of the IntToStr function, you can write real world tests with plain code.. just some code that runs through the functions and tells you success or failure. But in the case of more complicated functions or classes, such as web security, many times unit tests don't have an advantage over real world testing first.
Real World Web Browsers, Unit Testing?
Let's take another real world situation: HTML. How many times have you actually written "unit tests" for each html page you have on your website? The best way to test HTML is to fire open three popular browsers: Mozilla (firefox), Opera, Internet Explorer. That's your first priority and your first major test. That isn't a unit test, it is just a real world test (common sense). If you want to spend time writing unit tests for your HTML, you may end up on an infinite and boring joy ride. What is most important is that your HTML works in the popular browsers. You could go further and verify your html with the W3C compliance tool - a tool written by someone else, not you - hence not really a "unit test". Just a test tool. You could then go one step further and ask 10-100 people from different parts of the country/world to test and view your website. These are your top priorities. No where in your top priorities is "unit testing" since HTML is quirky with infinite amounts of variations. Unit testing might help a tiny bit - but there are higher priorities with HTML.
HTML however is just one example - this doesn't mean that unit testing isn't a high priority in all cases. Sometimes unit testing is more important than real world testing - such as if you are writing a University paper on a project that is all theory, but won't be released into the real world. Har Har Har.
Unit testing can be especially important in code that will be used by other programmers - for example a compiler needs to be rigorously tested for syntax and compile errors. The only problem is, the compiler can have an infinite amount of features - and some bugs may never need to be fixed, such as if no one ever uses a certain feature of the compiler for 10 years. Eventually after 10 years this bug may become important (let's say the GOTO statement wasn't working only if you were going to a label with a number/alphabit combination, but the GOTO label that was only alphabetic or only numeric worked fine). Sometimes writing unit tests and testing things that people will never use is a complete waste of time - and sometimes having your users or your developer-users write the tests is actually more effective than you thinking up and inventing the tests. For example, many of the Freepascal compiler tests are written by users who have discovered bugs in real world situations - and in many cases these are not unit tests but just real world tests, i.e. common sense: "Mr. Florian, the feature ABC isn't working, for example see this program I wrote which shows the compiler is broken". That's confirming a bug exists - is this a Unit Test, or is it a real world test, or is it just common sense that this is what you would do if you found a bug?
Obviously, testing things has been around for much longer than computers have ever been around. Someone just happened to coin a term around testing and write lots of wacky books about it. My term that I like to focus on is "Real World Testing" as opposed to "Unit Testing". If the real world tests happen to include unit tests, then so be it. But many times, real world tests are not source code based tests.
I haven't tested this Wiki for one year
When I wrote this wiki I wasn't worried about the quality of the product. I had so many other important things to get done. I wrote it in one day and then added some more features to it a few days down the road, and a few more features months later. It's a small source code snippet that runs this wiki - probably the smallest successful program I've ever written. I could write hundreds of tests and find hundreds of things to improve or fix. In fact infinite things. But the wiki just worked - in simplest form - and I uploaded it to the real world right away. As soon as I uploaded it to my server I started doing real world tests - such as injecting bad things into the URL variables. I didn't do unit tests, I did real world tests. My highest priority was to verify that the wiki was secure.
I didn't need to test it further with unit tests yet. Some day, I will test it - especially if I was releasing something more complicated to the public - as this wiki is extremely simple right now and private. But right now, unit testing don't matter one little bit. What was more important was that I started using the wiki and finding bugs in it from real world settings.
I found out that the Pragma No Cache or Meta No Cache needed to be set upon editing. Something I wouldn't have found with unit testing since only the browser could tell me this, as I temporarily forgot about no-cache issues during writing the web program. The only way for me to notice the no-cache issue would be to experience a real world situation where no-cache would cause issues - I couldn't write a unit test if I didn't have no-cache in my mind in the first place.
I also found a few possible potential security issues which ended up not being security issues after putting myself into cracker shoes (this due to the GetCgiVar function which I knew was already protected from XSS). But I updated the security just incase anyway, being overly cautious. Then a few months later some people actually tried to hack into the wiki more then ten times on different occasions.. nothing happened for them - they left out of boredom and moved on to the next server. I learned a bit more about the way they tried to hack into the system - again some real world experience and no "unit tests" done yet.
Something I really wouldn't have discovered with a Unit Test are the real world things that happened! This is important. Things like attempting to be a cracker with a web browser can't really be written inside a unit test - the unit test isn't a real enough environment. You could write a macro and have your web browser test your programs - but that isn't a unit test, that is just a real world browser test.
The no-cache issue I discovered couldn't really be discovered through a "unit test" as I would have had to know up front that there was a caching issue in the first place. Since I first had to be reminded of cache issue by testing my program in a real world web browser - not using a unit test. Trying to write a unit test that verifies caching is working isn't really possible since you need a web browser to visit the website and test that the cache works.. the only option would be writing a macro to have the web browser test the web program to see if caching works. Is that really unit testing, or is it now "real world testing using whatever tools suit the testing job"?.
In my case I fixed the cache issue once and never touched that code again for 1 year and still haven't touched it, nor do I see a need to go to the effort of spending 6 hours writing a cache test, especially since I can quickly just launch my browser and see if cache is working - and see if the meta no-cache and pragma no cache still exists in the header. I would immediately know if cache wasn't working since I enter text into the wiki almost every single day. I myself am the testing suite for my wiki - there aren't too many unit tests needed yet.
Real World Environments vs Partly Real Environments
Brute force real world testing is really what type of testing needs to be done on serious applications. Real world testing is more important than software based Unit Testing suites, except for special cases where unit testing can really emulate the real world environment close enough. Sometimes Unit Testing encourages hello world style or overly simplified tests to be written, since the programmer can easily forget to include or may find it possible to include many of the real world settings into a sheltered environment, the unit testing suite.
All succcesful software companies and successful open source projects hire or scout out real world testers (humans) to forcefully test the software for hundreds of thousands of real world problems - and this form of testing is far more effective and efficient than "unit testing", since the amount of users or hired testers usually outnumbers the amount of main developers. The testers and users of the software know the software and know what they want out of the software even more than the developers, and hence are better testers than software testing suites programmed by the developers. The users and testers of the software also are real people, real testers, and not just digital source snippets (which are in effect, unreal).
Sometimes it is best to fire up the software in real world environment, and test the hell out of it - even before "unit testing". If you can automate the real world testing by running your program through numerous tests, you aren't necessarily using "unit testing". Unit testing is source based testing - but the customers and users of the software applications are humans and not source code (except if the software being tested is a compiler, in which case Unit Tests act as the real world tests since all that needs to be done is code snippets need to be run through the compiler and tested - this is a special case where unit testing really is as effective or close to being as effective as the real world environment).
Although "unit testing" pinpoints the testing to smaller snippets of code, the "real world testing" on the other hand pinpoints the important real world problems! It is hard to tell what source to do more testing on, especially if the source isn't even used in the software application (some features are almost never used). One could spend days writing infinite unit tests on source code that is used only one or even zero percent of the time in the real world environment.
Having folks external from your development team test the software is important too (if at all possible), since other folks can find bugs that you and your development never would have anticipated - i.e. beta and alpha testing.
Basically, instead of a focus on "unit testing", developers should be focused on any "real world test" that will help the software project become better.
|
