How do I exec thee? Let me count the ways...

One of the strengths of Tcl is the ease of integration with other software, whether they be COM components, libraries or even executable programs that are not designed for interaction with other programs. Here we look the facilities Tcl offers related to the last of these -- running external programs and optionally interacting with them using standard I/O mechanisms.

The intent here is to compare at a high level the various commands Tcl offers for executing programs and highlight the distinguishing features of each. It is not to explain their use and working so I will not go into the syntax and details of each command -- that might be the subject of later articles.

We start off with the exec and open commands which are part of the Tcl language and portable to all platforms.

The exec command

The exec command starts a pipeline of one or more processes where the standard output, and optionally standard error, of each process is fed into the standard input of the next process in the pipeline. Normally the command waits for all processes in the pipeline to terminate and returns as its result the standard output of the last process in the pipeline. However, if the last argument to exec is &, the command returns immediately and the return value is a list containing the process ids of each process.

A simple example to list open connections using the Windows netstat and findstr programs:

% exec netstat -n | findstr ESTABLISHED
  TCP    192.168.1.128:49782    111.221.124.68:443     ESTABLISHED
  TCP    192.168.1.128:49791    111.221.124.119:443    ESTABLISHED

Alternatively, if we wanted to have it just save the output to a file and not wait

% exec netstat -n | findstr ESTABLISHED > connections.log &
%

Although the exec command is flexible in that it provides for multi-process pipelines, synchronous and background operation, per-process redirection of standard channels etc., there are some common situations where it is inconvenient or unsuitable.

The first situation arises when the executed program either produces a lot of data or produces it in incremental, continuous fashion. Because exec returns all output from the child process in one shot after it terminates, such programs can present a problem.

For example, the netstat command on Windows can be run to produce a list of network connections on a continuous basis by specifying a time interval.

% exec netstat -n 2

If you run this command, you will find it never returns because the exec continues to wait for netstat to complete which it never does. A similar issue arises with respect to passing data to the executed program. Various standard channel redirections can work around this limitation to some extent but do not provide a complete solution.

The open provides a more suitable alternative.

The open command

Like exec, the open command is a part of the core Tcl language. Although it is normally used to open a file, it can also be used to create a process. If the first character of the file name passed to open is |, the remaining characters in the file name are treated similar to the arguments for `exec. The command then returns a channel connected to the child process' standard input and output. The channel can then be read from or written to in incremental fashion.

% set fd [open "|netstat -n | findstr ESTABLISHED" r]
file4df3d0
% while {[gets $fd line] >= 0} { puts $line }
TCP    192.168.1.128:8862     65.55.68.103:443       ESTABLISHED
TCP    192.168.1.128:8863     91.143.88.180:5222     ESTABLISHED
...lines skipped...
% close $fd

You can even choose to set the channel to non-blocking and set event handlers to read and write to the channel, none of which is possible with the exec command.

Although the exec and open commands meet most needs for running external programs, there are some special cases on Windows that cannot be handled with them. The TWAPI extension provides the create_process and shell_execute commands for dealing with these cases.

The twapi::create_process command

The create_process command operates at a lower level than exec and open. It creates only a single process and unlike those commands, any desired pipelines have to be explicitly programmed.

What it does provide is more fine grained control of various aspects of the created process context. Some of the more useful ones include

• user account context, described in another post
• window attributes such as colors, position and size, title etc.
• attached console
• the environment variable values for the process
• the desktop and window station in which the process is created
• ability to create the process in suspended mode
• setting up security descriptors to control access to the process
• inheritance of open handles
• process priority
• process groups

As an example,

% twapi::create_process c:/windows/system32/netstat.exe -cmdline "-n 2" -newconsole 1 -background red -showwindow maximized

will start netstat in a new maximized console window with a red background.

The twapi::shell_execute command

The final command we describe for running programs is shell_execute. There are several uses for this command that are related to the Windows shell. We only summarize those related to program execution here.

The shell_execute command can be used to run programs similarly to exec or create_process. Like the latter, it cannot be used to run a pipeline of processes.

% twapi::shell_execute -path notepad.exe -params {sample.txt}

In addition, you can run a program indirectly by passing the command the path of a data file. This results in Windows running the program associated with the file type. This is useful when you need to run a "logical" program responsible for handling a file but do not know its path.

% twapi::shell_execute -path sample.doc

The above command will run the program associated with the .doc type, generally Microsoft Word. Something similar (with some limitations) can be done indirectly with exec or create_process as well by running the command shell cmd.exe and passing it the START command with the name of the data file. However, the shell_execute is not restricted to a "default" action on the data file. It can ask the program to take a specific action as well. For example, instead of opening the file, we can run the Word program to print it instead.

% twapi::shell_execute -path sample.doc -verb print

The action may not even be related to a file. For example, you can start the email program to compose a message

% twapi::shell_execute -path mailto:[email protected]

(Note: to be pedantic, this may not actually start a program if it was already running.)

Executing Windows programs in elevated mode

Windows Vista introduced the concept of running programs in elevated administrator mode. This is one task that can only be accomplished with shell_execute and none of the other commands. This is done by specifying runas as the action. For example, to run a command shell in elevated mode

% twapi::shell_execute -path cmd.exe -verb runas

will show the Windows UAC dialog asking permission for running cmd.exe in elevated mode.

In summary

So what command is to be preferred for running programs? A summary of the factors:

On Unix, or the code has to be portable, obviously exec and open are the only choices. Use the former for convenience unless the amount of data produced is large or will be produced in incremental fashion. In that case, use open, optionally with file event handlers.

On Windows, use the platform-specific commands when

• you need to run the process in elevated mode. Here you have to use shell_execute.

• you need to run under a different user context. Here you have to use create_process.

• running programs indirectly depending on data associations. Here shell_execute is not mandated but is the most convenient.

• you need to control the process environment, runtime, security and other settings. In this case create_process is the most flexible.

Otherwise, even on Windows, you are better off sticking to exec or open, certainly for pipelines but also for potential future portability if nothing else.