Automating VMware with VIX

Published , updated

As for many pieces of software, one of the issues in testing TWAPI is the number of different platform configurations under which the test suites need to be run. Manual configuration is tedious and error-prone so automation of the process using VMware and the vix package is a big win.

The Problem

Several aspects lead to a combinatorial explosion in the number of test configurations.

  • The operating system, starting with Windows XP and up to Windows 8.1/Windows Server 2012 R2
  • 32- and 64-bit versions of each
  • Service packs, which need to be tested separately because of differences in DLL exports
  • Tcl version 8.5 and 8.6
  • Multiple TWAPI distribution formats
  • Application suites (needed for testing some TWAPI functionality)

Even after triage, a significant number of combinations remain. For each configuration, the appropriate Tcl and TWAPI versions have to be loaded, test scripts run and results gathered.

This is where VMware and the Tcl vix package can be used to automate the process.

The Solution

The first step is setting up VMware virtual machines running the different versions of the operating system. I use VMware Workstation for testing but any VMware product can be used including the free VMware Player and VMware Server products. If using a product that supports virtual machine snapshots, you can make a snapshot corresponding to each service pack and save some disk space. For products like VMware Player which do not have the snapshots capability, separate virtual machines are needed for each service pack.

The above is a one time set up. Once that is done, test runs can be executed at any time, iterating through each configuration in turn. Each test cycle in the run then

  • Starts the appropriate virtual machine (VM) from a known state
  • Copies the selected Tcl version, TWAPI distribution and test harness to the target VM
  • Runs the test scripts, collecting and storing test results
  • Resets the VM to a known state so that persisted machine state changes do not carry over to the next test run

Each of these steps involves controlling the target VM in some fashion. VMware provides the VIX API for the purpose. We will make use of this via the Tcl vix package. Note that because this package is based on COM interfaces, it will only run on Windows host systems though the target VM may run any operating system.

The Demonstration

We will demonstrate the use of VIX through a sample tclsh interactive session. Of course, a real test harness will wrap these steps in a script. This session assumes you have the vix package installed as described in its documentation.

Starting the VM

We start off by loading the vix package, initializing it and connecting to the local VMware Workstation host. Because this matches the package defaults we do not need to specify any additional options. 

% package require vix
0.4
% vix::initialize
% vix::Host create host
::host

Having connected to the host, we open the virtual machine of interest. In the case of VMware Workstation, the machine is identified by the path to its VMX file. 

% set vm [host open "c:/virtual machines/vm0-xpsp3/vm0-xpsp3.vmx"]
::oo::Obj81::guest#1

It is a good idea to always start a test run with a clean slate so we revert the VM to a previously created snapshot called Base. 

% set snap [$vm get_snapshot Base]
::oo::Obj111
% $vm revert_to_snapshot $snap
% $snap destroy

We can then power up the VM. One thing to keep in mind when working with VMware VIX interface is that many commands depend on VMware tools to be running in the target VM. So after the VM starts up, we wait for the tools to finishing loading. Depending on the host system this can take a little time so we wait for up to a minute. 

% $vm power_on
% $vm wait_for_tools 60

Logging into the target VM

It has been straighforward so far but we now run into our first roadblock. Some TWAPI modules implement commands related to window management and the desktop. Testing these modules therefore requires a login session at the console and unfortunately there is no way to create one automatically with VMware. If there is a desktop already running, we can run programs in it but there is no way to create one.

Thus we have to manually log in to the VM (not shown here) before continuing with our next step. We assume we have done that here.

As an aside, the above is not strictly true because one can reduce security settings in the target VM so that manual logins are not necessary. I am loathe to lower security though even on test VM's.

Once we have manually logged into the console of the target VM, we return to our tclsh session to proceed with logging in through VMware as well. Because we want to interact with the desktop, we need to make sure we use the same account to login as we used for manually logging into to the target VM console. 

% $vm login tester myunguessablepassword -interactive 1

Copying files

We can now start off copying the files needed for testing. We will store everything under the c:\testdir directory. Note the various commands for file and directory management and transfer.

We begin recreating the directory structure and copying a single file Tcl executable that we use for testing. 

% $vm rmdir c:/testdir
% $vm mkdir c:/testdir/bin
% $vm copy_to_vm tclkit-cli-8.6.3-x86.exe c:/testdir/bin/tclkit-cli.exe
The file name is not valid

Oops. It turns out that VMware's VIX implementation will accept / as path separators in some commands but not others. Changing the path separator to \ (escaping it for Tcl's benefit) fixes the issue. You could also have of course used the file nativename to transform the path. 

% $vm copy_to_vm tclkit-cli-8.6.3-x86.exe c:\\testdir\\bin\\tclkit-cli.exe

The same command will also copy entire directory trees, and we use it to copy the twapi distribution of choice and test scripts. 

% $vm mkdir c:/testdir/lib
% $vm copy_to_vm twapi-bin c:\\testdir\\lib\\twapi-bin
% $vm copy_to_vm ../tests c:\\testdir\\tests

We are now ready to run our test scripts.

Running tests on the target VM

Running program on the target VM is very straightforward. For example, 

% $vm exec c:/windows/system32/notepad.exe -activatewindow 1
2684 0 0

The above would start notepad on the desktop and return its PID. The -activewindow option ensures the window is not iconized on start up.

We could use this method to run TWAPI's test suite wrapper but for pedagogical purposes, we will do things slightly differently. We will invoke our Tcl interpreter and pass it an inline script and wait for it to return. 

% $vm script c:/testdir/bin/tclkit-cli.exe {
    cd c:/testdir/tests
    lappend auto_path c:/testdir/lib
    package require tcltest
    package require twapi
    tcltest::configure -outfile results.txt
    source base.test
    exit 0
} -wait 1
2608 0 11
% $vm read_file c:\\testdir\\tests\\results.txt
base.test:  Total   456 Passed  435 Skipped 21  Failed  0
...remaining lines skipped...

In a real test driver, we would have used $vm copy_from_vm instead of $vm read_file to collect the results and store to a local file.

We are done with target VM and can shut it down and clean up. 

% $vm shutdown
% $vm destroy
% host destroy
% vix::finalize

The test runner script can now move on to repeating the sequence with the next configuration.

In Conclusion

Even if tests are fully scripted, (re)configuring the target platforms can still be a hindrance to running tests on a continuous basis. Automation using virtual machines is relatively simple to implement and removes much the tedium from the process. (Those not on Windows hosts can look at the vmrun program included by VMware as an alternative to the Tcl vix package albeit somewhat more inconvenient to program.)