Test only what you need to work
Bugs only exist in software because we as developers introduce them.
We don't mean to introduce them however there is no denying that they exist and that we've all come across them at some point in the software that we use. Thankfully there are a range of testing techniques that we can employ to ensure that the code that we produce is as bug free as is possible given the time that we have. The software development process is a much used phrase that describes the different parts involved in creating an application. Many development methodologies have grown and evolved over the years to control this process, these methodologies control how requirements are collect, designed (UI and code), development and tested. Application developers are instinctively drawn to the design and development steps of the lifecycle and can sometimes be guilty of forgetting about testing. However it is in the testing step that the first true indication of how successful the design and development steps has been, can be seen.
Testing is often given a bad press and is portrayed as being a manual process, that doesn't involve much creativity. However by thinking about how this functionality is going to be test while it is in design and development that it can result in us creating smarter applications that are more stable, resistant to unforeseen conditions and more extendable (maintainable).
Below we will cover what unit testing is, why unit testing is important, the other testing levels and what this means going forward.
What is unit testing?
Unit testing is ensuring that the smallest part (unit) of testable code in your application behaves as expected in isolation (from the other parts of your application). In object-oriented programming a unit is often a particular method within an application class, with the unit test (also being a method within a test class) testing one scenario (path) through that method. A unit test does this by providing a strict, written contract that the unit under test must satisfy in order to pass. If the unit has multiple paths through a method i.e. an if and else branch, then more than one unit test would need to be created to ensure that all paths where covered.
Unit tests are then combined into a test suite within a test target/project, this suite can then be run to give an increased level of confidence in that the application classes are valid.
Each unit test should be executed in isolation from other unit tests to ensure that the failure of a previous test to pass has no impact upon itself. It up to the unit test to ensure that any conditions (test data, user permissions, etc) that it needs to be present within the application are and to tidy up after itself when it is finished. This helps to ensure that the unit test is repeatable and not dependent upon the hosts environment. The unit test is also responsible for ensuring that the unit under test is isolated from other methods within the application and that any calls (relationship) it makes for information with other methods are mocked out. A mocked method will then return a known, preset value without doing performing any computation so that if a unit test fails we can have confidence that it has failed because of the code under test rather than having to hunt down the failure in it's relationships.
Depending on the methodology used on the project, the unit test may be written before or after the application unit and it may be by the same developer who will write the application unit or another.
Unit tests can be derived from the specification of the application and/or the application code itself, resulting in unit testing being a form of both black-box and white-box testing. Black-box testing is where test cases are designed based on the outputs from an application. White-box testings is where test cases are designed to take advantage of the internal implementation of your application's functionality ensuring that each path through a particular method is tested.
Does my project need unit testing?
This is a very common question and the simple answer is: No
There are any number of successful, stable and widely used applications available today on the app store that don't have any unit tests. These apps get to where they are because before each new version hits the store, the development team behind it manually tests the application to ensure that it works. As you can imagine at the start this did probably save time and effort because the application was small, containing only a handful of screens and very few paths between those screens. But consider the scenario that all app developers want to find themselves in: your app has taken the app store by storm and users love it, in fact they love it so much that they are emailing you to ask if your amazing app could do more. Excellent you think (rubbing your hands with glee), roll up your sleeves and get cracking, adding a new feature here and a new feature there, altering paths through existing screens and manually testing as you go. You release that version onto the store and even more success befalls you so much in fact that you decide to hire another developer to help you with the work load, you give this new developer some tasks to do and he sets about doing them however he doesn't know the app as well as you do so he doesn't know that variable isRunning in class A has a knock on effort in class B and decides to alter it to better fit the new feature that was introduced into class A. When it comes to testing you now discover a bug, the app is crashing, after chatting with the new developer you decided to have a look around and eventually find the isRunning alteration, making the change to class B to get the up working again. The time it takes to manually test the app is increasing with each new version but that's fine because there are now two developers working on the app so you split the testing effort and keep going. The next comes with a range of new features, that you split between yourself and the new developer, during the process the new developer notices that isRunning, isn't behaving the way that he programmed it to so he updates it and continues on with adding the new features. Because of the amount of time its now taking to manually test your app and because the competition is becoming fierce and you need to still ahead of the game, you decide the cut the amount of testing and test in your ability's as a developer to write perfect code first time so you push to the store. Before long you start noticing 1 star reviews along the lines of:
“App crashes all the time, avoid!!!”
Armed with that detailed bug report, you spend the next 4 days trying to track down is causing your app to crash, all the time more and more 1 star reviews are being posted. Finally on the fifth day, (with only strand of your sanity remaining) you discover that isRunning that you resolved weeks ago has been the cause of it all. You make the fix (sack the new developer) and push the new version to the app store. Problem solved, the apps working again however you've just spent a lot of goodwill credit that you had with your users and damaged the app in the eyes of new users.
I know what you're thinking that the above scenario could easily have been overcome with better communication (or proper encapsulation between class A and B) and yes it could have however the ever present danger of relying solely on manual testing would still have been there and eventually a bug similar to that one would have made it onto the app. However the trouble with manually testing an app is that as the application grows the amount of time you need to test the application grows. As human beings we are great at completing unique and interesting tasks however we are not great at performing tasks when they get repetitive, we become distracted, jump around, forget steps, misread steps and day dream. Manually testing an application is a repetitive task, especially if this the eighth version of your application and you are checking a part of the application that shouldn't have changed. It is at this stage that the opportunity for you to miss a step and decide that some piece of functionality is working correctly (as expected) when in fact it is not. However as we've been told for years, computers love repetitive tasks. It's what they are built for and they have become very quick at completing these tasks. By creating a unit test suite, we can push these more mundane and repetitive tasks to the computer and have them run through the test suite every time we push a new commit to the repository or are thinking about uploading a new version to the App store and leave the more interesting testing to us, such as is path A as easy as it could be, should I introduce a button there, etc. One nice side effect of creating a test suite is that we get a greater sense of confidence in the stability of the app and can more aggressively (quickly) add new functionality or edit existing functionality as the test suite takes only seconds/minutes to execute rather than the hours/days that a manual test suite could take. And if we discover a bug that wasn't covered in the test suite, we can add it and guard ourselves against it in the future so that the isRunning bug won't bite us again.
It's important to keep in made that a unit test suite, is not a cure-all pill that will make your app bug free and an overnight success. It's entirely possible to create a test suite that isn't very good, is incomplete or is only concerned with testing that the word "goat" has four characters, twelve hundred times. How extensive your test suite is, will depend upon the aims of the project and what resources you have to implement the test suite. It will take time to prepare, create and maintain your test suite, you will find that writing your test suite will impact upon how design project and most importantly it will take time to develop the mindset needed to write high quality test suites.
When should a unit test be written?
An effective way of writing unit tests is to write the unit at the same time as writing the method (unit) under test. Depending on your style and the methodology being followed, the writing of the unit test itself can be directly before or after the method. Writing your unit test(s) before you write the method can seem strange at first, as the method won't exist (maybe even the class) and you are using the unit test as a form of design. However this strangeness is actually an advantage, as you begin thinking about the method from a unit testing point-of-view. A unit testing point-of-view differs from a programming point-of-view in that when testing you are most concerned with producing code that is: Isolation and Cohesion, whereas often when programming you are more concerned purely with a working outcome. If your thinking:
“Wait a moment, I always take the time to properly design my methods/classes”
then that's amazing and you're doing a great job. However that probably isn't always true for everyone on your team (or even you all the time). We can use unit tests to help a programmer think more about the quality of the code he is producing by making him instantly have to consume that method himself. So if a method makes extensive use of other method calls inside itself to get the data it needs to do its job, we us unit testers will need to mocked out those method calls to ensure that they don't fail themselves. A better approach from a unit testing (time) point-of-view. would have been for the method to require all the data it needs up front as parameters. It then becomes much easier to pass these parameters into the unit directly from our unit tests rather than creating a range of mock classes that e.g. only return string values. This new method design as the advantage of decoupling our method from our methods and making it a lot more maintainable. The same is true for a method that is doing too much which often presents itself as having too many paths through it. To fully unit test that method we would need to test each path which means writing more unit tests and increasing the chance that we accidentally forget to cover a path through (this path will due to sod's law, be bug ridden and destroy your outstanding app store reputation). A better approach would be to break the method up into smaller more focused units of work that can each be addressed individually, allowing you to better focus on covering each path.
Writing your unit tests directly after you write the method, can result in a less taxing thinking process as you have something concrete to write the unit test against however as you write the unit test you may discover that the method isn't as broken down as it could be and so have to refactor the method you just created to make unit testing an easier job.
Writing unit tests when the method itself is written will make for an easier process as the method's structure and propose will be fresher in your mind than if you wait to write unit tests.
Who should write a unit test?
Ideally writing unit tests should not be viewed a different task from the writing the method itself but rather that the method isn't completed until it passes all its unit tests. Because of this tight coupling between unit test and method, the best person to write the unit test is the developer who's responsible for the method itself. The developer has most knowledge of how the method is structured and what the method's outcomes should be so writing the unit tests should be a quicker process for them than for anyone else writing it. Remember a unit test is a form of both black-box and white-box testing so writing your unit test solely against the specification for that method can mean that you miss some special cases in the method. A drawback to the developer writing the unit test is when the dreaded programming blindness sets into a method, it could spread to the unit tests. Programming blindness is where some code contains bugs or has missing functionality however the developer who wrote it can't see these failings. A reaction to this blindness can lead to an organisation implementing a structure where one developer writes the method and another writes the unit, this structure can result in better quality unit tests however often the time penalty for doing so outweighs the benefit. A better approach is to implement code reviews to improve the developers programming ability by looking through the mistakes that were created and help overcome this blindness. It's important to remember that writing effective unit tests is a skill on its own and needs to be practiced at before you can become “good”at it. Don't allow initial setbacks to derail you (the benefits are too great!).
Other testing Levels
Testing comes in many levels from unit testing to acceptance testing and everything in between (and beyond). As developers we will focus on unit testing however before we can begin our journey into unit testing we need to understand (briefly) the different levels of testing that exist, the benefits of these levels of testing (and any downsides) and who typically performs these testing activities.
Integration testing
Integration testing is the level up from unit testing. Integration tests are responsible for ensuring that the relationship (interface) between units of functionality behave as expected. Integration tests often take the same structure of unit tests but without mocking that out the relationship under test. Because integration testing is focused on the interaction between two methods rather than the internal workings of each method, integration tests can be derived entirely from the specification and so its possible for someone other than the developer who wrote the methods to write the integration test.
System testing
System testing is the level up from integration testing. System tests are responsible for ensuring end-to-end functionality and non-functional requirements. System tests generally comprise of business processes or typical working that the system should be able to handle. Typically system testing is conducted by dedicated testers and have very little input from the developers. This is the level of testing that is most often associated with manual testing via the interface with the testing environment (as closely as possible) mirroring that of the end user's system. Failure to mirror the end user's environment may lead to a situation where the system works on the tester's device but not of the end users. System testing is most effective when created by a formal test team.
Acceptance testing
Acceptance testing is the level up from system testing. Acceptance tests are responsible for ensuring that the app meets the requirements and that the app can be accepted as functionally complete by the app's stakeholders. There is often a level of overlap between acceptance and system testing. Typically acceptance tests are carried out by a formal test team, other stakeholders and invited end-users. An area of acceptance testing that most people are aware of is: alpha and beta testing.
Alpha
Alpha testing involves the end user coming to development site and using the software in a controlled environment where the end user would use the app to perform working set of tasks. This is very useful for discovering misinterpretation in the requirements documents or discovering new paths through the app that had not been considered before.
Beta
Beta testing is perhaps the most wide known form of testing. It involves releasing the software to a sample group of end users on their own site and having them use the software as they would during their normal working day.
The testing levels above are by no means an exhaustive list of testing levels available and some of you may even disagree with me about what each level involves or the order they flow in. I'm ok with this and wish you no harm. The important point to note here is that by creating unit tests that doesn't mean that other levels of testing don't need to performed. Unit testing exists within an ecosystem of other testing levels, with each level building confidence in the app.
What do you think? Let me know by getting in touch on Twitter - @wibosco