The Skeptical Tester

[article]
Summary:

Testers are people who ask questions, think critically about the answers, and then ask more questions—repeatedly. Fiona Charles reminds testers that their success depends on maintaining a healthy skepticism.

 Someone asked me recently what is the worst mistake a tester can make when starting out on a project. I didn’t have to dig very hard for my reply: “Believing what you’re told, and accepting what you’re given.” OK, that’s two mistakes. They’re inseparable.

Do I mean that every tester should go around vibrating with suspicion and saying, “Don’t give me that!”? No, of course not. That would be neither professional nor practical.

I mean that you should never assume that what you have been told is the whole truth and nothing but the truth. Never believe that the documents you’ve been given contain all the information you will need to absorb and understand for testing the system. Never take as fact anything you haven’t taken trouble to question and, if possible, verify. Where you can’t verify and you need to move on, make working assumptions and then revisit them periodically to see if you’ve since learned anything that would prove or disprove them.

Testers are people who ask questions. We are skeptical people who examine critically all the information that comes our way and then go looking for what’s missing. We know critical questioning is essential to our work because the information we need is often scattered, buried, or not really known by anyone on the project. And sometimes projects nurture myths—commonly held beliefs that can obstruct, yet may collapse when probed.

On one big program where I was program test manager, everyone knew that all test personnel must have a particular certification. It was in the contract with our government client. My manager told me this when I started (although I possess no certification and they engaged me), and all my leads believed it. I was concerned that this requirement  would get in my way, but I had bigger problems to deal with and at least I seemed to be getting good functional testers.

But then I needed performance testers. Week after week I met with the hiring manager only to learn that we still hadn’t got the skilled people my performance test lead urgently required. I discovered that our recruiter was routinely turning away highly qualified performance testers on the sole ground that they weren’t certified.

This was crazy. Even in markets where most testers have certifications, performance testers typically don’t bother to certify. If we really were contractually bound, we would need to seek an exemption from the client. So I asked one of my leads to search our large and complex contract for the certification clause.

When finally unearthed, the clause read, “certification or equivalent experience.” So much for received belief! When we stopped acting on myth, we had no trouble recruiting the skilled people we needed.

On another project, I almost hit a showstopper through believing what I was told. To test the end-to-end integration of a new e-commerce site, I needed to have a coordinated dataset across a big suite of applications. None of the many planning, supply chain, or merchandising systems upstream from the online store was being changed for the project. We just needed to get product, pricing, and promotions data from them in the end-to-end test. The teams responsible for those systems were used to testing together, and they assured me they had a coordinated product set for use in all their integrated testing. They’d be ready to participate in the end-to-end test when I needed them.

Great! I focused my efforts on other urgent tasks. The online order management system and warehouse management system were new, and there were myriad associated changes to the many systems downstream, including backend financials.

But some time later, I learned that the product set with which the e-commerce site was going to launch contained mostly items that the upstream systems lacked in their test data. Preparing a new coordinated dataset for the upstream systems would be both time-consuming and expensive. The managers of those teams balked at taking on the work, yet the end-to-end test was considered critical for successful launch of the strategic e-commerce site.

Ultimately, I found a way to segregate the upstream systems and their existing data without compromising the critical flows in the end-to-end test, but it was a close call. I hadn’t asked enough questions early enough about the data, and it could have killed my test.

At two other clients, I was told that the general ledger (GL) system was out of scope for the project, and hence for the end-to-end test. It’s understandable that managers would stipulate this. The more systems you test with, the greater the cost. Why include the backend financial systems if nothing there has changed? I didn’t accept this in either case, and I gently probed until I confirmed my suspicions.

User Comments

13 comments
Anonymous's picture
Anonymous

Read the **** contract isn't just a matter of pulling a fast one on a naive client. It can be crucially important to make sure we understand what is really required. I've been amazed at how clear and reasonable contractual requirements can get transformed by a process of Chinese Whispers into something that is going to be a nightmare to comply with. If it seems too bad to be true, check the contract! it might just get you off the hook.

May 21, 2012 - 1:43pm
Anonymous's picture
Anonymous

Read the **** contract isn't just a matter of pulling a fast one on a naive client. It can be crucially important to make sure we understand what is really required. I've been amazed at how clear and reasonable contractual requirements can get transformed by a process of Chinese Whispers into something that is going to be a nightmare to comply with. If it seems too bad to be true, check the contract! it might just get you off the hook.

May 21, 2012 - 1:43pm
Anonymous's picture
Anonymous

Read the **** contract isn't just a matter of pulling a fast one on a naive client. It can be crucially important to make sure we understand what is really required. I've been amazed at how clear and reasonable contractual requirements can get transformed by a process of Chinese Whispers into something that is going to be a nightmare to comply with. If it seems too bad to be true, check the contract! it might just get you off the hook.

May 21, 2012 - 1:43pm
Anonymous's picture
Anonymous

Read the **** contract isn't just a matter of pulling a fast one on a naive client. It can be crucially important to make sure we understand what is really required. I've been amazed at how clear and reasonable contractual requirements can get transformed by a process of Chinese Whispers into something that is going to be a nightmare to comply with. If it seems too bad to be true, check the contract! it might just get you off the hook.

May 21, 2012 - 1:43pm
Robert Robredo's picture

I'm a new online tester or quality checker online hired via a freelance site. I have read that you have an experience with order management system software; I'll be working on one soon too (link is my project site.) I'd like to ask for some tips on how can we be assured that I am hired legally, will be paid after my services and that the company is not a fraud?
Thank you in adv for your advice. I'm still new in this freelance world, and unsure how the system works especially for QC or online testers.

August 31, 2012 - 7:39am
Robert Robredo's picture

I'm a new online tester or quality checker online hired via a freelance site. I have read that you have an experience with order management system software; I'll be working on one soon too (link is my project site.) I'd like to ask for some tips on how can we be assured that I am hired legally, will be paid after my services and that the company is not a fraud?
Thank you in adv for your advice. I'm still new in this freelance world, and unsure how the system works especially for QC or online testers.

August 31, 2012 - 7:39am
Robert Robredo's picture

I'm a new online tester or quality checker online hired via a freelance site. I have read that you have an experience with order management system software; I'll be working on one soon too (link is my project site.) I'd like to ask for some tips on how can we be assured that I am hired legally, will be paid after my services and that the company is not a fraud?
Thank you in adv for your advice. I'm still new in this freelance world, and unsure how the system works especially for QC or online testers.

August 31, 2012 - 7:39am
Robert Robredo's picture

I'm a new online tester or quality checker online hired via a freelance site. I have read that you have an experience with order management system software; I'll be working on one soon too (link is my project site.) I'd like to ask for some tips on how can we be assured that I am hired legally, will be paid after my services and that the company is not a fraud?
Thank you in adv for your advice. I'm still new in this freelance world, and unsure how the system works especially for QC or online testers.

August 31, 2012 - 7:39am
Fiona Charles's picture

Hi Robert,

You're right to be cautious -- freelancers do occasionally get ripped off. However, I'm not a legal expert and I suggest that if you're really concerned, you start by getting a legal opinion of your contract. You should also find out what "legal hiring" means in your jurisdiction. The laws that govern regular employees may not cover free-lancers.

Here are a couple of general tips that it's a good idea for any freelancer to practice.

Before signing anything, research the reputation of the company proposing to hire you. Ask around if you have any doubts about them.

Read every page of the contract carefully. Don't be tempted to skip over stuff that looks like boilerplate. If you do decide to sign, initial every page even if you haven't been asked to do that.

I don't know any way to assure that you will actually get paid. But don't let your customer snow you. If the money dries up and they miss a scheduled payment, stop working immediately until you get paid. (They might miss the first payment through bureaucratic sloppiness. That's not usually a major concern. But missing a later payment is a danger signal. Don't get trapped into working for free!)

Go out and buy a copy of "Secrets of Consulting" by Gerald M. (Jerry) Weinberg. It has lots of wonderful advice for contractors as well as consultants.

Most of the time, it all goes well and we have good mutual relations with our customers.

Best of luck with your freelance career!

Fiona

August 31, 2012 - 6:30pm
Fiona Charles's picture

Hi Robert,

You're right to be cautious -- freelancers do occasionally get ripped off. However, I'm not a legal expert and I suggest that if you're really concerned, you start by getting a legal opinion of your contract. You should also find out what "legal hiring" means in your jurisdiction. The laws that govern regular employees may not cover free-lancers.

Here are a couple of general tips that it's a good idea for any freelancer to practice.

Before signing anything, research the reputation of the company proposing to hire you. Ask around if you have any doubts about them.

Read every page of the contract carefully. Don't be tempted to skip over stuff that looks like boilerplate. If you do decide to sign, initial every page even if you haven't been asked to do that.

I don't know any way to assure that you will actually get paid. But don't let your customer snow you. If the money dries up and they miss a scheduled payment, stop working immediately until you get paid. (They might miss the first payment through bureaucratic sloppiness. That's not usually a major concern. But missing a later payment is a danger signal. Don't get trapped into working for free!)

Go out and buy a copy of "Secrets of Consulting" by Gerald M. (Jerry) Weinberg. It has lots of wonderful advice for contractors as well as consultants.

Most of the time, it all goes well and we have good mutual relations with our customers.

Best of luck with your freelance career!

Fiona

August 31, 2012 - 6:30pm

Pages

About the author

Fiona Charles's picture Fiona Charles

<span class="Text"><strong>Fiona Charles</strong> is a Toronto-based test consultant and manager with thirty years of experience in software development and integration projects. Fiona is the editor of <em><a href="http://www.stickyminds.com/s.asp?F=S1149_BOOK_4" target="_blank">The Gift of Time</a></em>, featuring essays by consultants and managers of various professions about what they've learned from Gerald M. Weinberg. Through her company, Quality Intelligence, Inc., Fiona works with clients in diverse industries to design and implement pragmatic test and test management practices that match their unique business challenges. Her experiential workshops facilitate tester learning by doing, either on the job or at conferences. Contact Fiona via her Web site at <a href="http://www.quality-intelligence.com/" target="_blank">www.quality-intelligence.com</a>.</span>

AgileConnection is one of the growing communities of the TechWell network.

Featuring fresh, insightful stories, TechWell.com is the place to go for what is happening in software development and delivery.  Join the conversation now!

Upcoming Events

Nov 09
Nov 09
Apr 13
May 03