Currently there's a discussion going around the Blogosphere and Twitterverse about whether or not driving development with acceptance tests–particularly automated acceptance tests--is a good thing. Many expert practitioners have weighed in, including Gojko Adzic, George Dinwiddie, Ron Jeffries, and James Shore.
I find value in each perspective, and I know what works for my team, because we've been experimenting with different approaches for developing software for more than six years. If your team hasn't already figured out how to make sure you develop the code the customers want--at the quality level they need, in a timely manner, and all that good stuff--how do you make sense of all these different techniques? Most importantly, if you aren't enjoying your work now, how can you find more reward in what you do?
A Learning Culture
Personally, I think it starts at the top. Is the business really committed to making changes to address pain points and improving software quality? Or, are some managers just trying the latest development fad so they can look like they're trying to take charge?
It's funny to me when I hear people use the term "waterfall" when what they really mean is "hopelessly dysfunctional process where no good practices are in place." I've worked on a waterfall team with full test automation from the unit level on up, continuous integration, and constant collaboration between the technical and customer teams. We produced some of the best-quality software I've ever seen. If people have the wrong motivations or are lazy and undisciplined, I don't care what process you follow, it's going to keep failing. So, "agile" isn't the one right way either. It just tries to put the focus where it belongs--on getting good people and letting them do their best work.
Let's assume management is willing to implement true change and to provide the time and resources for it to happen. Change has to start with a team's soul-searching. First, who should be on the team? With the "Whole Team" approach, everyone involved in developing the software--including testers--is on the development team, and they all take responsibility for quality and testing. I've seen the most success with this approach, but it comes with a caveat: Each team member must be seen as having equal value.
Next, the team has to agree on values. It's easy to say, "We are committed to delivering the best-quality software we can," but what does that really mean? If the team is going to use excuses such as "We can't do TDD because of the architecture" or "We can't do CI because our product is too complex," the team is not really committed to quality. Every team member must take responsibility for quality and for completing the testing activities to ensure it. Developers hate touchy-feely conversations like this, but change won't happen without identifying the values and principles that will guide that change. The team has to agree that they won't get stuck–that they will find a way around every obstacle.
So, let's say management values quality over speed and nurtures a learning culture where mistakes are tolerated and experimentation is expected. The team has established its values about their work and product. Now, the team can start taking baby steps to improve their development practices and processes. At every step, they look to their team values and principles to help decide what practice to implement or what tool to try.
"We're feeling rushed, but if we don't stop and refactor this test design now, we will pay for it later with higher test maintenance cost."
"Testing is always squeezed to the end of the iteration. What can we as a team do to address that?"
"We misunderstood that requirement. How can we make sure we're on the same page as the customers?"
Your team values light the way.
It might take many months for the team to feel like they've turned the corner and have significantly improved their product's quality, but I bet the customers notice a difference pretty quickly.
One Right Way
I can't tell you the one right way to test and develop software. If you don't have the enlightened management I've described above, try to enlighten them. Ask them to identify the biggest pain points and work with you to experiment with ways to address those. If you don't have the motivated teammates eager to take responsibility for quality and testing, try leading by example. Show what you do with your own passion for doing good work.
The one right way for your team to code and test will continually evolve over the years. In fact, I'm sorry to tell you that you'll never find the one right way; it's a moving target, but you'll get closer and closer to it.
Please read all these blog posts and as many books as you can. Participate in as many online and local user groups as possible. Keep an open mind. Bring the ideas and techniques you learn about back to your team. If one seems likely to help your team overcome a particular obstacle, give it a try. If it doesn't work, you learned something. If it does, celebrate and move on to the next obstacle.