JV: Okay, cool.
MA: But yeah, that ended up being a big thing for us, so it was a more fluid process. We weren’t pre-generating a ton of work that we would throw away, and it put the information where everyone was looking. I remember not to long after we started doing this that I had one director’s company and say, “This job is amazing, like I’ve never seen this much information about how things get tested and what was broken.” I’m like, “That’s what we do all the time. Look at the test plans that you asked us to generate.” But just really bringing that information right into their faces where they saw it on a daily basis, that was a huge win for communication throughout the team.
JV: Yeah, and that seems like it’s not an excessive amount of test documentation. If you just put out that plan, it’s something you could easily read and notice right there. It’s not written in document jargon.
MA: Well, sometimes people want kind of like an old-fashion test thing, because that information, you just suck it up and put it in a CSV file. So, yeah, it was powerful. The other thing we decided was that our regression tests were going to be automated and that was just going to be the end of the story. I was pretty much done with manual, and, probably, in waterfall we didn’t have a lot of automation here. We [did] a regression pass to two to three weeks as we went through and made sure everything was celled.
If you take that two or three weeks from every product release you can say, “Let’s take all that energy and put it into test automation.” You get all your regression test happening instead of doing all that manual time. I had to make every person on my team a developer.
MA: Yeah. I had ten people on my team and at the end of it all of them could code, all of them could build test for their various automated systems.
JV: And some people didn’t have any experience coding before?
JV: No. All right. Well, what better way to learn than on the job.
MA: Yeah. We did a whole lot of work to make sure that they had access to training programs; code school is great. But yeah, that was a big thing to do. We had to free up time for them to be able to learn, because it was really important to me that we didn’t say, “Okay, here’s your book. Go home at night and read all this and by tomorrow…” People need to have their free time.
JV: You also said you were rebuilding the automated testing infrastructure.
JV: What does that entail, besides work?
MA: It kind of goes into the efficiency bucket again. When I first started my team, within the first couple weeks I had them kind of step back from their exploratory testing and start focusing on test automation. One of them was very tiny; it was only three of us at that point and one of them wasn’t much of a coder and the other one was a pretty decent coder and it’s like okay, we need to figure out how we’re going to even automate these products. So I gave them two weeks each to just go and explore the problem, like what are the test frameworks we can use all that jazz.
MA: Not being coders, in some cases they started building test frameworks on top of existing tools. And when they tended to code a lot of it wasn’t modular code; a lot of it wasn’t reusable code. It worked and it did the job, but to get things up and running quickly they took shortcuts; like having for our system as kind of this backend database that we need to talk to and they would have a single database instance up on the internal network and all the tests would run against that. Then they would work very hard to make sure that they only looked at the data they created.
We quickly got into the situation where our tests were depended on previous runs, because they were looking at a test twenty minutes into. They were looking at something that got started in the first test, but we couldn’t test that one test when it broke; you had to do a full twenty minute run. And talking to people, a lot of people run like that. It’s so hard to do data management when you’re doing automative testing and to be able to do it quickly that. And again, they’re also in a situation where they don’t have a huge development team either, because they hired customers.
JV: I mean that just seems like a huge issue nowadays. You got data management and testing.
MA: Yeah. Well, if you do something like Rails, Rails has a whole great system built in to handle it where you can actually set up flag in your test. You’re like “Hey, stick this all in memory for the next ten commands and then you can pop it and it lets it all go.” So you can kind of get this machine into a state, and I saw that and I really like it. I had one of my folks build a data management system for us and we had a central web service that we could do the request and say “Hey, build me a data base in this state and at this IP with this version and this data set,” and it would automatically do it and set it up in this nice clean bucket for us and hand us back a URL.
And for all my test automation we started going through and ripping out all the bits that we were relying on previously existing states, and just having these nice clean test cases that were running against these test instances that spin up on the fly; we to be able to do the for exploratory testing too, because we have a web interface for the whole thing. So someone could go in and say “Oh man, an automative test fail. That’s weird. I want that exact same state—that same version, make me one,” and then they can connect and do it by hand.
JV: And you’re still using the system now?