Multi-user, Multi-process Test Automation

The utility fork() allows us to create new processes and have those processes run concurrently. If we have sudo rights, every forked process can run as a different user in a different environment.

Find the User's Stuff with PATH
Both unix-like systems and Windows have an environment variable called PATH. PATH is simply a list of directories. The PATH on my Mac (which is a unix-like OS) is

/usr/local/bin:/usr/local/bin/ruby:/bin:/sbin:/usr/bin:/usr/sbin

The PATH on a Windows machine is found in Start/Control

Panel/System/Advanced/Environment Variables

Whenever a user launches a program with a single word, like "ruby" or "notepad", the OS tries to find the particular program in the list of directories in PATH. For my testing, I need to have the correct copy of the st-admin utility found in the PATH of the user running the tests.

My Test Launcher
My test launcher has to do three things:
 

1. It has to send HTTP requests to a port. The port is related to the euid of the current user.
2. It has to find the command-line script "st-admin" in the PATH of the particular user running the test.
3. It has to launch st-admin actions. The proper environment in which st-admin is to run is determined by the euid of the process that calls the st-admin utility.

A Little Code
Here are some examples of how to look at what we've discussed so far, with user "qa11" as an example:

#echo $PATH
/home/qa11/stbin:/home/qa11/bin:/hom
e/qa11/local/bin:/usr/local/bin:/home/qa11/src/st/current/nlw/bin:/home/qa11/src/st/current/nlw/dev-
bin:/home/qa11/src/st/trunk/nlw/bin:/home/qa11/src/st/trunk/nlw/dev-
bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/bi
n/X11:/usr/games

Effective userid (euid)
#perl
my $euid = $>;
print "$euid = ¥n"
>> 2010
#ruby
euid = Process.euid
puts euid
>> 2010
#shell
id
uid=2010(qa11) gid=2010(qa11)
groups=4(adm),2010(qa11),65532(machine

A Little Code

My test-launcher has these methods:

                   get_testcases
                   set_users
                   set_ports
                   set_failure_port
                   add_tests_to_queues
                   run_all_tests

Most of these methods set up data for the tests to run. They're pretty boring until you get to run_all_tests. Here is a simplifed version of my code:

@queues = ['cases1', 'cases2', 'cases3', 'cases4']
@ports = ['22209', '22210', '22211', '22212']
@users = ['qa9', 'qa10', 'qa11', 'qa12']

def run_all_tests

            0.upto(@queues.length - 1) do
|index|

                  @queue = @queues[index]

#FOR EVERY QUEUE OF TEST CASES

               @env_port = @ports[index]

               @dev_user = @users[index]
               fork do

#FORK A NEW PROCESS

                  Process.euid = 0

#BE ROOT...

                  fork_euid =
@env_port[1..4]

                  Process.euid =
fork_euid.to_i

#...SO WE CAN SET THE euid FOR THE FORKED PROCESS

                      path =

'/home/dev_user/stbin:/home/dev_user/bin:/home/dev_user/local/bin:/usr/l

ocal/bin:/home/dev_user/src/st/current/nlw/bin:/home/dev_user/src/st/cur

rent/nlw/dev-bin:/home/dev-

user/src/st/trunk/nlw/bin:/home/dev_user/src/st/trunk/nlw/dev-

bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/bin

/X11:/usr/games'

 #SET A TEMPLATE PATH VARIABLE

                  ENV['PATH'] = path.gsub!(/dev_user/,@dev_user)

#REPLACE THE dev_user VARIABLE IN PATH WITH THE REAL USER

                  @queue.each do |@testcase|

#RUN EVERY TEST IN THE QUEUE INSIDE THE FORKED PROCESS

                     def run_test
                        puts "running #{@testcase}
on #{@env_port}"

                        puts "effective
user id: #{Process.euid}"

                        puts "user name:
#{@dev_user}"

                        puts

                        @content = `~/stbin/run-wiki-tests
--timeout

50000 --test-server "http: // machine.socialtext.net:#{@env_port}"
--test-

workspace test_data --plan-page "#{@testcase}" 2>&1`

#SHELL OUT TO RUN THE TEST ITSELF

                     end #def run_test

                     run_test

#RUN EACH TEST IN THE QUEUE

                  end #queue.each

                  end # fork do
               end # queues do
               end #run_all_tests

Make it Fast

Ten years ago the few tools that existed for testing at the UI were poorly made and very expensive. They worked, but just barely--some of the time. (Remember, first you have to make it work.):

Open source UI test tools, like SAMIE, then Watir, then Selenium, raised the quality and lowered the cost of UI testing. (After you make it work, then you make it good.)

As testing tools become more powerful and achieve better quality, it's becoming expected