Can you really have it all? An in-house dream team that can flexibly and seamlessly reshape itself to meet any change in your software landscape, rising to any development, testing, or problem-solving challenge-while maintaining all your current projects, timelines, and clients? Well, sure you can...sometimes. But realistically there will be times in your career when you may need to look outside your organization for help with some (or all) of your work.
How do you go about making informed decisions to ensure success with outsourcing? How do you choose the right provider, and then maximize their effectiveness? And as the project progresses, how do you demonstrate that you've gotten what you've paid for-that the provider is delivering what you said you needed? During my experiences with outsourcing, I've learned some important lessons:
With an understanding of these five guidelines, you can avoid several of the pitfalls commonly encountered when outsourcing pieces of your software effort. The more care you give to analyzing your needs and understanding the ability of the outsourcing company to meet them, the more satisfaction you can generate for your company and your customers.
Do Your Homework
Those of us who work in a hectic and often chaotic software environment typically do not have enough time to make informed decisions about day-to-day activities-let alone involved projects like outsourcing. Doing careful analysis up front will minimize headaches down the road.
Before you even start looking for outsourcing companies, it's important to look at your own company. Are your company's goals and objectives clearly defined? For example, if you're planning to outsource your testing effort, what is the quality standard of the software to be shipped? What are the release criteria? Does a written, signed-off test plan exist? A member of your test team can help you define these targets for your company and its customers. And the importance of documentation cannot be overstated. By ensuring that your company has fulfilled its obligations, you won't be overly dependent on the outsourcing company for guidance and direction.
Once you've addressed these questions in house, you can start looking for an outsourcing company. View this as you would the process of interviewing a candidate for a position with your company: pre-screening, interviewing, and checking references. Knowing what your needs are, what job you need them to do, and whether they have the capability to do that job are equally important. Performing due diligence and following a process help to form a lasting working relationship between the two parties that will pay off in the long run.
Your requirements are going to guide your evaluation criteria in the pre-screening phase. Some important factors to assess include:
(These factors will come into play in more depth during your interviewing stage, but thinking about them now in your pre-screening phase may help you weed out some of your weaker candidates.)
Once you've identified your top few contenders, it's time to contact them and start preliminary dialogues about your needs and their capabilities. Don't shortchange the amount of time you budget for this evaluation phase. The time you invest with the vendors in these early stages-starting the planning process-will mean that neither party will be caught off-guard or blindsided by future demands.
As you evaluate candidates, seeing is believing. If possible, plan on visiting their company, meeting some people, and touring their facilities. How quickly can they mobilize to start on your project? How much lead time do they require? Don't rush your way through that onsite visit, and don't leave until all of your questions are answered, and you have a strong sense of who they are and what they can do for you. Don't be shy; you're performing an assessment of their capabilities-just as you would in a job interview-so you'll have sufficient data to compare them to other competitors. Try to review the job descriptions and résumés of key workers to get a sense of their backgrounds and strengths. Ask to see samples of their work products and status reports. (With test projects, for example, be sure to look at their test lab, test plans, and test scripts/cases.)
After you've identified your requirements, interviewed the candidates, and had all your questions answered, it's time to make a decision.
One useful tool for choosing the right provider is a Kepner-Tregoe Matrix (see Figure 1) that lists your outsourcing requirements/criteria down the left column and each vendor that you're evaluating across the top row. Before you start comparing vendors, decide on the importance of each of the criteria, listing those values in each Importance column. Then, for each vendor, rate the extent that they meet your criteria-inserting those scores in the Rating column. Finally, take the product of each Importance and Rating and list the results in the Total column. Adding your totals will give you a Grand Total for each vendor-with the highest score pointing to your best choice. By keeping the criteria limited and focused to your "must-haves" you'll avoid any overcompensation with the "like-to-have" and "nice-to-have" criterion.
As part of this process, be sure to ask for references and talk to a few of their past and current customers. There's nothing like a little reputation research with their customers to obtain a final decision-making point.
During the decision-making process, you may conclude that outsourcing is not an option for you at this time. Maybe this level of expense is beyond your budget, or the costs outweigh the benefits. Presenting outsourcing as a business case will at least demonstrate to Upper Management that you've done your homework.
This may all sound like a lot of work, but I have experienced firsthand that skipping the up-front analysis doesn't save time. On one outsource testing project, time-to-market pressures cut down on the number of vendors we felt we could spend time with or evaluate. (It's always good to compare alternatives, and reducing our options put us in a sort of blind spot.) To make things worse, we didn't view their facilities prior to signing the agreement because neither we nor they knew which of their testing locations was going to be used for our work.
Other than making a couple of reference calls, we really hadn't done our homework. While the outsourcing effort was satisfactory, it certainly could have been better. Had we assessed other vendors and dug a little deeper we would have had greater success on the project-and with fewer workarounds.
Know What You're Getting Into
If you did a good job with your homework, this part will be much easier. Here is when you need to hash out all of the outsourcing details and make sure that all the bases are covered. Once you've selected your preferred vendor, you will need to sign some documents with them, such as a Confidentiality/Nondisclosure Agreement (NDA) and a Statement of Work. Read through these documents carefully, making sure you understand the terms.
Keep in mind that everything is negotiable prior to both parties signing any document, so now is the time to obtain clarification, make agreed-to changes, and ensure that you're getting what you asked for. Once the ink has dried, it's a done deal. While you can always add additional outsourcing scope later (via an Engineering Change Order, for example), wouldn't it be better to get it right the first time? In particular, pay attention to the following items:
Also consider the following: How busy is the outsourcing company? How much attention and priority will they be giving your project-and your team? Depending on whom you choose-and when-the quality of the outsourced effort may vary from vendor to vendor. The quality of your outsourcing effort may be affected if they are understaffed and/or overbooked. Can they devote what's required to ensure that your outsourcing effort is a success?
Another point to keep in mind is that while you might be looking to the vendor for expert advice, there are limits to their knowledge and experience. You, as the customer, should know what you want and remain in the driver's seat. The vendor is just there to help you along your way. Avoid excessive obedience to or dependence on the outsourcing company.
In addition, see if you can specify the exact site where your work will be performed, especially if you can choose a location that is close to your company. The default decision is that the vendor will load balance your outsourcing effort to whoever is available based on their current workload (not necessarily their geographical proximity to you). If you're able to select a vendor whose location is within driving distance from your company, you'll be able to periodically visit them to monitor their progress. Make sure that they're receptive to this idea (in the contract) so that you can visit them at any time for any reason. You are the customer, remember?
A project or team lead who's responsible for managing the outsourced staff is a good idea. You want to have at least one main point of contact that you can communicate with to obtain the status of the outsourced software effort. Frequent communication with this person can help alleviate several concerns-and can help you remain confident that the outsource staff remain dedicated and focused on your project's goals.
Most likely you will go through one or more revisions of the agreements before both parties sign. It's a good idea for your legal department to check these documents before you sign them. Outsourcing may be one of the few times during your career when having a legal expert review these documents will pay off, so here's where you can get it right the first time.
Inevitably you will encounter delays at almost every level, not just with the outsourcing company, but with your company as well. Involving the vendor as early as possible will help reduce delays in having the outsourcing company start when you'd like them to.
Depending on your software's complexity, the outsourcing vendor may take weeks just to install and learn to use it. Here's another incentive for you to do your homework when evaluating the capabilities of the people who will be putting their hands on your software. Until you sign a contract you won't have a guarantee that you'll have your favorite people assigned to your project, but it's worthwhile to go ahead and determine the overall skill level of the vendor's staff.
Are you and your vendor in the same time zone-or in the same country? Geographic issues may impact the outsourcing effort. As the distance increases, the delays you can expect with communication (e.g., phone, email, fax) also increase. Waiting a half a day for replies to your email can be commonplace. One thing that helped during our outsourced testing was establishing preset communication times corresponding to when each party would be available to talk. This was very effective in dealing with the issues surrounding time zone differences.
In our case there were other time challenges as well. We encountered delays with contract signing, work start dates, accounts payable, and more...such as when we tasked the vendor to obtain some servers with older operating systems (e.g., HP/UX, OSF/1, AIX). Not only did they have difficulty finding vendors that rented these systems and waiting for the deliveries, but the outsourcing company itself-not having extensive knowledge or experience with these operating systems-further delayed our schedule because they had to spend time learning these operating systems. With better planning we could have eliminated some of these delays.
Our outsourcing company admitted that our Web-based application was a "9 out of 10" on their difficulty scale. Because of that, our staff had to spend more time helping the vendor get up to speed and educating them on our product, thus causing delays with the project schedule. Their learning curve with understanding our product tacked on additional project time that we had not anticipated.
When my organization outsourced testing, the issue of defect management was one area of early noticeable disconnect between us and our outsourcing vendor. Part of the problem was that because we didn't know who was going to be testing our software, we couldn't target our training or impart our product knowledge to the testers. This reduced their testing effectiveness and further delayed our schedule. Moreover, early on our staff had to deal with duplicate bugs being reported by the vendor and entered into our defect tracking system. We suspected that this was because they had a junior tester assigned to our project who initially didn't search our bug database for duplicate issues, and had a longer learning curve for our product.
Constantly Involve the Outsourcing Company
Another lesson I learned was to constantly find ways to involve the outsourcing company in our daily activities. I discovered that the more they became part of our team-and not just a separate extension of our project-the better results we had. This eliminated the "us vs. them" mentality that sometimes disenfranchises in-house and outsourced teams.
In addition to integrating the outsourcing company into your team, ensure that they follow your internal processes and procedures. Both parties will have their own way of doing things, but for the project you'll need to decide on one way (which may involve a hybrid of both party's processes). Procedures for bug tracking, reporting status, and other tasks should be noted in the contract.
As you create and maintain your schedule of work and deliverables, you'll need updates from the outsourcing company's staff. Their assumptions, dependencies, risks, and time estimates affect your project schedule and should be incorporated into your schedule.
We gave the vendor's staff their own login and password for our defect tracking system so they could input issues directly. We made sure that they noted who was entering the issue so that we could track down that person for additional information if need be.
We also made sure that the outsourcing representatives called in to our weekly staff meetings, not just as observers, but also as active participants. They were frequently asked questions regarding the defects they were finding, their perspective on the state of the software, and any issues that they noted. I was particularly interested in the metrics they were reporting and the critical defects they detected.
If the outsourcing company had been located within a reasonable drive, I would have periodically visited them and checked on their activities. This would have helped to bridge the geographic gap. Mentally, I treated them like a separate SQA team in another location. Partly because of that, teamwork, trust, and synergy took longer to develop. Moreover, I would have welcomed the occasional visit by them to our company to partake in our staff meetings in person as well as assist us with some of our testing challenges.
Analyze the Results
How do you know if you are getting what you paid for? Are you satisfied with the outsourced results? These questions should be answered on an ongoing basis throughout the outsourcing project. Upper Management will want to know if the business case for outsourcing is worth it.
Based on the signed contract, the vendor should be providing you with periodic (e.g., weekly) status reports that you can incorporate into your reports. The metrics they're reporting should be stated in the signed contract and should match the metrics that you're tracking. If you're outsourcing testing work, for example, you and the vendor should both be measuring your testing results in identical ways, and following the same product metrics (e.g., the number of defects that are open, critical, fixed, and found, as well as the find/fix ratio and the number of defects by build).
One source of evaluating and understanding the results will come from the vendor's deliverables, which in testing projects could include test plans, test cases/scripts, defect reports, Requests for Enhancements (RFEs), status reports, and invoices.
Pay particular attention to the invoices: who is billing, at what rate, and for what? I made sure that there were full names (in addition to their titles) listed on the invoices along with their agreed-to hourly rate. I kept up with the invoices to catch any errors (and there were some) as soon as possible so that the vendor could get paid within their contracted timeframe. In addition, you should review their status reports carefully to assess the results of the work they're doing.
What additional results can you analyze? How do you measure success with the outsourcing effort? Success for your company may mean that the project was on time, on budget, and/or within scope. One way to track this is in terms of money and time. You can note this in your status report with planned vs. actual matrices (see Figure 2, previous page). On a week-by-week basis, keep track of the amount that you planned (Dollars Budgeted/DB) and the amount that you spent (Dollars Spent/DS). By calculating the ratio (DS/DB) between the Dollars Spent and Dollars Budgeted, you can see if the outsourcing effort is going as you'd planned-or if it's spinning out of control. A similar approach can be taken with the amount of time budgeted and spent.
Even though the outsourcing effort may be on time and on budget, what about the quality of the work? You can measure this attribute in two dimensions: effectiveness and efficiency. Ensure that your outsourcing providers are not only doing the right things but doing things right. Define the important performance (success) criteria by which you'll judge the outsourcing company. The process of analyzing an outsourcing company's results can be compared to the process of conducting a performance review for one of your employees: set goals, monitor progress, and evaluate outcomes.
Some things to consider with regard to your provider's effectiveness are:
Some things to consider with regard to your provider's efficiency are:
One way to verify the quality of their effort is to perform periodic assessments or audits of their work habits and their work products. Careful examination of the outsource effort may be the best way to assure that you're getting the quality you're expecting. The visits that you make to their facilities, the documentation that you inspect, and the people that you speak with will provide tangible evidence of the quality of their work, and thus the overall outsource effort. Because the outsourcing company may juggle its staff from project to project, it may be impossible to evaluate individual performance-so stick to analyzing the outsource effort as a whole. See if they're transferring their knowledge to your team members. Determine how well they're fitting in with your team's culture. Talk to your team and find out what they think of the outsourcing effort-and if they would want to work with them again in the future.
Outsourcing can be a great way to augment your software efforts. Making the time and effort to effectively manage the process of identifying, negotiating, and monitoring the outsourcing effort will boost your chances of a successful outcome. Doing some good comparative research up front and analyzing the results on an ongoing basis help to justify the business case of outsourcing. One benefit of hiring the services of an outsourcing company is that the next time you require their services, you'll have a lot less homework to do, and you'll have a good sense of what you're getting into. After the initial engagement, stay in touch with the outsourcing company and remember to give them as much lead time as possible for future engagements. They will enjoy this competitive advantage and you'll enjoy more peace of mind next time around.