I did finally get an agreement on writing a book on application test automation—one I had wanted to write for years already, as automated testing doesn't appear to be a love at first sight topic for many. And, like testing in general, in many implementations, it tends to be subordinate to requirements specifications and application coding, be it a project or a product. And, hell, who really loves testing? It is not typically what the average developer gets enthusiastic about. When raising the question during many of my workshops on automated testing, it often even takes a while for functional consultants to raise their hand.
Inevitably, I did pose the question to myself: do I love testing? And I answered it with a yes. A big YES. This subsequently led to additional questions, such as: what makes me love testing? Did I always love testing? Why do I love it while the rest of the world seems not to? Questions with answers that makes me go around the world evangelizing test automation, stubbornly sharing my findings, and pushing Microsoft to improve their tests to make them better and reusable. And all because I reckon that it is fun—big fun!
Having been an app tester in the former Dynamics NAV Global Development Localization (GDL) team at Microsoft, I surely got exposed to the testing virus. You could say that I had to learn to do the testing job, as I was paid for it. But it also suited me well, with me apparently having this specific DNA kind of thing that makes testers what testers are. Having a pride in breaking the thing and, meanwhile, loving to prove, hopefully, its robustness. And, not in the least, daring to break it, running the risk that your developer will no longer like you.
One afternoon at Microsoft, my developer team member walked into our office and stopped next to my desk. Him being very tall and me sitting, I had to turn my head up to look at him. "What's up?" I asked. "Don't you like me anymore?" he responded. Me: ? Him: "Don't you like me anymore?" Me: "Nope, still like you. Why?" Him: "You rejected my code; don't you like me anymore?" Me: "Dude, I still like you, but concerning your code, my tests show it somehow useless."
Testing is not rocket science. Nor is automated testing. It's just another learnable skill. From a developer's perspective, however, it requires a change of mindset to write code with a totally different purpose than you are used to. And we all know that change is often not the easiest thing to achieve. Because of this, it's not unusual for attendees at my workshops to get to a certain level of frustration.
Application testing is a mindset and it needs a big dose of discipline too—the discipline to do what needs to be done: to verify that the feature was built right; to verify that the feature under test meets the requirements—and the discipline when bugs are reported and fixed to execute the whole test run again, and again, with any new bug, to ensure that the verification is complete.
I tend to see myself as a disciplined professional. I have been quite a disciplined tester, with a high rate of bug reporting. But, did I always love testing? You know, in those days, all our tests were executed manually, and with each bug I found my discipline was challenged in some way. Imagine my mind running when executing the fifth test run after another bug fix. It's 4:00 PM and I am almost done. At breakfast, I promised my wife that I would be home on time for whatever reason. Let's pick a random one: our wedding anniversary. So, me being a disciplined tester, my promise to be home on time, 4:00 PM, and … I … hit … another … bug. Knowing that fixing it and re-running the tests will take at least a couple hours; how do you think my mind is running? Right: binary.
I had two options:
- Reporting the bug would keep me at work and make my wife highly disappointed, to say the least
- Not reporting the bug would get me home on time and save a lot of hassle at home
Had automated tests been in place, the choice would have been quite simple: the first option, resulting in no hassle at home and no hassle at work.
In this chapter, we will discuss the following topics:
- Why automated testing?
- When to use automated testing.
- What is automated testing?
Plainly said: it all boils down to saving you a lot of hassle. There are no emotions or time-intensive execution will keep you from (re)running tests. It's just a matter of pushing the button and getting the tests carried out. This is reproducible, fast, and objective.
To clarify, automated tests are the following things:
- Easy to reproduce
- Fast to execute
- Objective in its reporting
If it was as straightforward as that, then why haven't we been doing this in the Dynamics 365 Business Central world all those years? You probably can come up with a relevant number of arguments yourself, of which the most prominent might be: we do not have time for that. Or maybe: who's going to pay for that?
Before elaborating on any arguments, pro or con, let me create a more complete why not? list. Let's call it the whys of non-automated testing:
- Costs are too high and will make us uncompetitive.
- Dynamics 365 Business Central platform does not enable it.
- Customers do the testing, so why should we bother?
- Who's going to write the test code? We already have a hard time finding people.
- Our everyday business does not leave room to add a new discipline.
- There are too many different projects to allow for test automation.
- Microsoft has tests automated, and still, Dynamics 365 Business Central is not bug free.
Dynamics 365 Business Central is not as simple as it used to be way back when it was called Navigator, Navision Financials, or Microsoft Business Solutions—Navision. And the world around us isn't as one fold either. Our development practices are becoming more formal, and, with this, the call for testing automation is pressing on us for almost the same reasons as the why of non-automated testing:
- Drive testing upstream and save costs
- Dynamics 365 Business Central platform enables test automation
- Relying on customers to do the testing isn't a great idea
- Having a hard time finding people—start automating your tests
- Test automation will free up time for everyday business
- Keep on handling different projects because of test automation
- Automated tests are code too
Regarding the costs, I am inclined to say that, on average, the Dynamics 365 Business Central project goes 25% over budget in the end, mainly due to after go-live bug fixing. I am not going to elaborate much on who's to pay for this, but my experiences are that it's often the implementation partner. The math is quite simple if assuming that to be the case. If you're spending 25% extra on your own account at the end of the line, why not push it upstream and spend it during the development phase on automated testing, building up a reusable collateral?
During my time at Microsoft in the 2000s, research had been performed on the cost of catching a bug in the different stages of developing a major release of a product. If my memory is not failing, the cost of catching a bug after release was found to be approximately 1,000 times higher than when catching the bug already at requirement specification.
Translating this to the world of an independent software vendor (ISV), this might roughly be a factor 10 lower. So, the cost of catching a bug all the way downstream would be 100 times higher than all the way upstream. In case of a value-added reseller (VAR) doing one-off projects, this could be another factor of 10 lower. Whatever the factors would be, any spending upstream is more cost effective than downstream, be it more formalized testing, better app coding, code inspection, or more detailed requirements specifying.
In all honesty, this is a no-brainer, as this is the topic of this book. But it is worthwhile realizing that the testability framework inside the platform has been there ever since version 2009 SP1, released in the summer of 2009. So, for over nine years the platform has enabled us to build automated tests. Does it sound strange if I say that we have been sleeping for all that time? At least, most of us.
I agree that customers might know their features the best, and as such, are the presumable testers. But can you rely 100% that testing isn't squeezed between deadlines of an implementation, and, in addition, between deadlines of their daily work? And still, in what way does their testing contribute to a more effective test effort in the future? How structured and reproducible will it be?
Posing these questions answers them already. It isn't a great idea, in general, to rely on customers testing if you want to improve your development practices. Having said that, this doesn't mean that customers should not be included; by all means, incorporate them in setting up your automated tests. We will elaborate on that more later.
At this very moment, while I am writing this, all implementation partners in the Dynamics world are having a hard time finding people to add to their teams in order to get the work done. Note that I deliberately didn't use the adjective right in that sentence. We all are facing this challenge. And, even if human resources were abundant, practices show that, with growing business, in either volume or size, the number of resources used does not grow proportionally.
Consequently, we must all invest in changing our daily practices, which very often leads to automation, using, for example, PowerShell to automate administrative tasks and RapidStart methodology for configuring new companies. Likewise, writing automated tests to make your test efforts easier and faster. Indeed, it needs a certain investment to get it up and running, but it will save you time in the end.
Similar to getting the job done with proportionally fewer resources, test automation will eventually be of help in freeing up time for everyday business. This comes with an initial price, but will pay off in due time.
A logical question being posed when I touch the topic of spending time when introducing test automation, concerns the ratio of time spent on app and test coding. Typically, at the Microsoft Dynamics 365 Business Central team, this is a 1:1 ratio, meaning that for each hour of app coding there is one hour of test coding.
Traditional Dynamics 365 Business Central implementation partners typically have their hands full of customers with a one-off solution. And, due to this have dedicated teams or consultants taking care of these installations, testing is handled in close cooperation with the end-user with every test run putting a significant load on those involved. Imagine what it would mean to have an automated test collateral in place—how you would be able to keep on servicing those one-off projects as your business expands.
On any major development platform, such as Visual Studio, it has been common practice for a long time already that applications are delivered with test automation in place. Note that more and more customers are aware of these practices. And more and more, they will be asking you why you are not providing automated tests to their solution.
Each existing project is a threshold to take having a lot of functionality and no automated tests. In a lot of cases, the major part of the features used is standard Dynamics 365 Business Central functionality, and, for these, Microsoft has made their own tests available since version 2016. Altogether, over 21,000 tests have been made available for the latest version of Business Central. This is a humongous number, of which you might take advantage and make a relatively quick start on. We will discuss these tests and how you could get them running on any solution later.
There is no way of denying it: automated tests are also code. And any line of code has a certain probability of containing a bug.
Does this mean you should refrain from test automation? If so, it sounds like refraining from coding in general. For sure, the challenge is to keep the bug probability to a bare minimum by making the test design a part of the requirements and requirements review, by reviewing the test code like you would want to review the app code, and by making sure that tests always end with an adequate number of verifications by putting them under source code management.
Still not convinced on why you would/should/could start using test automation? Do you need more arguments to sell it inside your company and to your customers?
Here are some of the arguments:
- Nobody loves testing
- Reduced risks and higher satisfaction
- Once the learning curve is over, it will often be quicker than manual testing
- Shorter update cycles
- Test automation is required
Well, almost nobody. And, surely, when testing means re-running and re-running today, tomorrow, next year, it tends to become a nuisance, which deteriorates the testing discipline. Automate tasks that bore people and free up time for more relevant testing where manual work makes a difference.
Having an automated test collateral enables you to have a quicker insight than ever before into the state of the code. And, at the same time, when building up this collection, the regression of bugs and the insertion of new ones will be much lower than ever before. This all leads to reduced risks and higher customer satisfaction, and your management will love this.
Learning this new skill of automating your tests will take a while, no doubt about that. But, once mastered, conceiving and creating them will often be much quicker than doing the thing manually, as tests often are variations of each other. Copying and pasting with code is ... well ... can you do such a thing with manual testing? Not to mention re-running these automated tests or the manual tests.
With agile methodologies and cloud services, it has become normal practice that updates are delivered with shorter time intervals, leaving even less time to get full testing done. And Dynamics 365 Business Central is part of this story. If Microsoft isn't forcing us in, our customers will be requesting this from us more and more. See also the preceding discussion.
Last but not least on this incomplete discussion of argument on the why of test automation, and perhaps the sole reason for you reading this book: automated tests are required by Microsoft when you are going to sell your Dynamics 365 Business Central extension on AppSource. It is also great for me to be able to share my great passions with you: being a speaker on various conferences; conducting workshops; writing this book.
All joking aside, this is a great opportunity for all of us. Yes, we're pushed, but we have been lingering on this topic for too long. Now, let's get on with it.
Nevertheless, you might rightfully wonder whether test automation is the silver bullet that will solve everything. And I cannot deny that. However, I can tell you that, if exercised well, it will surely raise the quality of your development effort. As mentioned before, it has the following benefits:
- Easy to reproduce
- Fast to execute
- Objective in its reporting
Those are enough arguments to convince you why you would want to use automated tests, I guess. But how about when to use them? Ideally, this would be whenever code is changed to show that this functionality, already having been tested, is still working as it should, to show that recent modifications do not compromise the existing application.
This sounds logical, but what does this mean when you have no automated tests in place? How do you go about start creating your first ones? Basically, I would advise you to use the two following criteria:
- What code change will give the highest return on investment when creating automated tests?
- For what code change will your test automation creation improve your test coding skills the most?
Using these two criteria, the following kind of code changes are typical candidates for your first efforts:
- After go-live bug fixing
- Buggy code
- Frequently modified code
- Business-critical code being changed
- Refactoring of existing code
- New feature development
- Microsoft updates
An after go-live bug reveals an omission in the initial test efforts that can often be traced back to a flaw in the requirements. Frequently, it has a restricted scope, and, not the least important, a clear reproduction scenario. And by all means, such a bug should be prevented from ever showing its ugly face.
You have this feature that keeps on bugging you and your customers. Bugs keep on popping up in this code and it never seems to stop. The most elementary thing you should start with is the after go-live bug fixing approach as previously discussed. But, even more importantly, use this code to create a full test suite for the first time.
One of the basic rules of good code governance is that code should only be changed when it is going to be tested. So, if code is modified frequently, the consequence is that it will also be tested frequently. Automating these tests will give a good return on investment for sure.
Thorough testing should always be the case, but, given circumstances, it is unfortunately not always doable. Testing changes made to business-critical code, however, should always be exhaustive. You can simply not afford any failure on them. Make it a point of honor to find even the two to five percent of bugs that statistics tell us are always there!
Refactoring code can be nerve-racking. Removing, rewriting, and reshuffling. How do you know it is still doing the job it used to? Does it not break anything else? It certainly needs to be tested. But, when manually done, it is often executed after the whole refactoring is ready. That might be already too late, as too many pieces got broken. Grant yourself peace of mind and start, before any refactoring, by getting an automated test suite in place for this code to prove its validity. With every refactor step you take, run the damn suite. And again. This way, refactoring becomes fun.
Starting from scratch, both on test and app code, will be an irrefutable experience. For some, this might be the ultimate way to go. For others, this is a bridge too far, in which case, all previous candidates are probably better ones. In Section 3, Designing and Building Automated Tests for Microsoft Dynamics 365 Business Central, we will take this approach and show you the value of writing test code alongside app code.
Incorporating any update from Microsoft, be it on-premises or in the cloud, your features must be (re)tested to prove they're still functioning like before. In case you do not have automated tests in place, begin creating them. Do this based on the analysis of the various changes and the risks they might entail of introducing errors.
We discussed why you might want to automate your tests and when to do this; or more specifically, where to start. But we didn't spend any thoughts on what automated testing is. So, let's do that before we conclude this chapter.
With automated testing, we address the automation of application tests, scripting manual application tests that check the validity of features. In our case, these are the features that reside in Dynamics 365 Business Central. You might have noticed that we have been using somewhat different terms for it:
- Test automation
- Automated tests
- Automated testing
These all mean the same thing.
What is exploratory testing? Check out the following link for more information:https://en.wikipedia.org/wiki/Exploratory_testing.
On the other hand, they are complementary. Manual testing will still contribute to raising the quality of a feature, making use of creative and experienced human minds able to find holes in the current test design. Automated testing might also include so-called unit tests. These are tests that verify the working of atomic units that altogether make up a feature. Typically, these units would be single, global AL functions – units that would never be tested manually.
More on unit and functional tests can be found at the following link: https://www.softwaretestinghelp.com/the-difference-between-unit-integration-and-functional-testing/.
In this chapter, we discussed why you would want to automate your application tests, and when you might want to create and run them. And we concluded with a short description of the what—what is automated testing?
In Chapter 2, The Testability Framework, you will learn what technical features are built into the Dynamics 365 Business Central platform that enable us to run and create automated tests.