Using TLS connections from Tcl

Published 2021-02-19

The use of the Transport Layer Security (TLS, formerly known as SSL) is becoming increasingly prevalent to secure network communication, particularly with the browser and search companies pushing web sites to move to HTTPS instead of HTTP. This post discusses the options for using TLS over Tcl sockets and specifically some important potential pitfalls to keep in mind with respect to certificate validation. The discussion is limited to client-side operation.

There are three packages available for Tcl, all binary extensions, that make TLS available from Tcl. You can build them from the sources listed in References. They are also included in most binary distributions of Tcl.

The TclTLS extension provides the tls::socket command with an interface that is a superset of Tcl's socket command. The returned channel can then be used as a secure transport for any application protocol. The package is portable across multiple platforms.
The TWAPI package includes the equivalent tls::socket command. Unlike TclTLS, this package is Windows-specific but has advantages in terms of certificate management as detailed later.
The TclCurl package provides a Tcl interface to the famous curl multiprotocol library. Unlike the previous two, this does not provide Tcl channels to be used for communication. Rather, the interface provided is the application protocol level similar to Tcl's built-in http package. However, curl implements excellent support for a very wide variety of application protocols.

The packages are loaded in the usual fashion:

(src) 4 % package require tls
1.7.22
(src) 5 % package require twapi
4.5.2
(src) 6 % package require TclCurl
7.74.0

Basic operation

The basic syntax for creating client connections for both TclTLS and TWAPI is similar to Tcl's socket command. The returned channels can then be used in standard fashion. The session below illustrates the basic usage to retrieve a Web page. For convenience, we first define a httpreq procedure that sends the HTTP GET method on a specified channel.

proc httpreq {so host} {
    fconfigure $so -translation crlf
    puts $so "GET / HTTP/1.1"
    puts $so "Host: $host"
    puts $so "Connection: close"
    puts $so ""
    flush $so
}

A plain unencrypted HTTP request with standard Tcl sockets is made simply as

(src) 12 % set so [socket www.example.com 80]
sock000002E4984D7C70
(src) 13 % httpreq $so www.example.com
(src) 14 % read $so
HTTP/1.1 200 OK
Accept-Ranges: bytes
Age: 597261
... Additional lines omitted ...

A request using TclTLS is almost identical except for the connection to port 443 instead of 80.

(src) 15 % set so [tls::socket www.example.com 443]
sock000002E4984AE8C0
(src) 16 % httpreq $so www.example.com
(src) 17 % read $so
HTTP/1.1 200 OK
Age: 563946
... Additional lines omitted ...

So also for TWAPI.

(src) 20 % set so [twapi::tls_socket www.example.com 443]
rc0
(src) 21 % httpreq $so www.example.com
(src) 22 % read $so
HTTP/1.1 200 OK
Accept-Ranges: bytes
Age: 507635
... Additional lines omitted ...

Both commands accept the same options as the Tcl socket command though that is not illustrated here.

NOTE: As an aside, there is a slight difference in behavior between tls::socket and twapi::tls_socket. The former establishes a TCP connection and returns immediately. The TLS negotiation is not done until the first attempt at data transfer or until the tls::handshake command is called. On the other hand, twapi::tls_socket command only returns (assuming synchronous mode) after TLS negotiation is completed.

The pattern for TclCurl is different from TclTls or TWAPI because it does not deal with raw TCP channels. Rather its interface is at the application protocol level. To retrieve a web page, we specify the URL (which also embeds the application protocol) and the name of a variable in which to store the retrieved content.

(src) 29 % curl::transfer -url https://www.example.com -bodyvar body
0

The return value of 0 indicates success and body contains the content.

(src) 30 % set body
<!doctype html>
<html>
<head>
    <title>Example Domain</title>

    <meta charset="utf-8" />
... Addition lines deleted ...

On return, the variable body contains the actual HTTP body and not the HTTP headers but the latter is available by also specifying the -headervar option with the above command.

NOTE: For simplicity, in this post we are using the curl::transfer command which encapsulates multiple curl functions and is a wrapper around the curl::init command which has innumerable options and flexibility.

Certificate validation

A TLS connection provides for both privacy as well as authentication that the remote end is who they claim to be. The latter is accomplished through the validation of the certificate provided by the remote end. This validation may be the default one used by the package or through a callback provided by the application. In our above examples, we did not provide a validation callback which meant the default validation was in use.

It is important to understand the significant differences between the default validation implemented by the various packages. The badssl.com web site provides some test URLs for this purpose. For example, the wrong.host.badssl.com site provided a certificate whose subject does not match the host while the expired.badssl.com site provides an expired certificate as hinted at by the name.

Validation in TWAPI

Let us start with the TWAPI package and attempt to connect to some URLs with invalid certificates.

(src) 39 % twapi::tls_socket expired.badssl.com 443
The received certificate has expired.
(src) 40 % twapi::tls_socket wrong.host.badssl.com 443
The target principal name is incorrect.

As expected (and one would think desired) both connections fail with an error. This is because TWAPI uses the Windows native SChannel implementation for TLS which in turn uses the certificate store on Windows to verify received certificates.

There are times when one wants to loosen these validations. For example, many sites use self-signed certificates, either for cost reasons or laziness (since free SSL certificates are now available and cost cannot be an issue), where the root signing certificate is not trusted.

(src) 41 % twapi::tls_socket self-signed.badssl.com 443
The certificate chain was issued by an authority that is not trusted.

Connections to these sites will fail by default because the root signing certificate is not in the Windows trusted root certificate store. Yet, this may be a legitimate site (using a corporate internal root certificate for example) that the application needs to connect to. In such cases, you can specify your own verifier callback for validating the server's certificate.

In the simplest case, the verifier can always return true. As seen below, the connection now completes successfully.

(src) 47 % proc verifier_true {args} {return true}
(src) 48 % set so [twapi::tls_socket -verifier verifier_true self-signed.badssl.com 443]
rc8
(src) 49 % httpreq $so self-signed.badssl.com
(src) 50 % read $so
HTTP/1.1 200 OK
Server: nginx/1.10.3 (Ubuntu)
... Additional lines omitted ...

In the general case though, you do not want such a blanket disabling of validation. Instead the verifier is passed a certificate context which can then be explicitly validated using the twapi::cert_verify command. This command offers a wide range of fine grained options for controlling the validation. Suppose for instance you only want the untrusted root errors to be ignored but not expired certificates etc.

proc verifier_ignore_untrusted_root {so context} {
    set cert [twapi::sspi_remote_cert $context]
    set result [twapi::cert_verify $cert tls -ignoreerrors unknownca]
    twapi::cert_release $cert; # IMPORTANT - release cert to free memory
    if {$result eq "ok"} {
        return true
    } else {
        return false
    }
}

This procedure can be passed in to twapi::tls_socket as the command to be used for verifying certificates.

(src) 55 % set so [twapi::tls_socket -verifier verifier_ignore_untrusted_root self-signed.badssl.com 443]
rc10
(src) 56 % httpreq $so self-signed.badssl.com
(src) 57 % read $so
HTTP/1.1 200 OK
Server: nginx/1.10.3 (Ubuntu)
Date: Fri, 19 Feb 2021 09:16:56 GMT
... Additional lines omitted ...

This works as desired; self-signed certificates are accepted. On the other hand, expired certificates still cause a failure, again as desired.

(src) 59 % set so [twapi::tls_socket -verifier verifier_ignore_untrusted_root expired.badssl.com 443]
SSL/TLS negotiation failed. Verifier callback returned false.

The above is still simplistic for pedagogic purposes. In reality, you should make the checks more stringent than just accepting any root which is just almost as bad as having no validation at all. For that, the twapi::cert_verify offers a full range of options including building your own trusted root store. A full description is too long for this post.

Validation with TclCurl

Attempting to access the URLs with invalid certificates using curl gives the following results.

(src) 61 % curl::transfer -url https://expired.badssl.com -bodyvar body
35
(src) 62 % curl::transfer -url https://wrong.host.badssl.com -bodyvar body
60
(src) 63 % curl::transfer -url https://self-signed.badssl.com -bodyvar body
60

Instead of 0 (success), the command now returns a Curl error code indicating validation failure. Again, this is desired default behavior.

NOTE: This default behavior may depend on the validation engine in use. See below.

If this desired behavior is to be overridden to not do any certificate validation, the -sslverifypeer option may be set to 0. The command will then succeed, ignoring any certificate validation errors.

(src) 68 % curl::transfer -url https://self-signed.badssl.com -bodyvar body -sslverifypeer 0
0

NOTE: The documentation says the value must be 0 to disable validation. It appears this is exactly the case and any boolean false value will not work.

There are other options related to TLS in curl, too many to describe here. Moreover, many options depend on specific validation engine in use.

The one thing you have to be aware of with curl is that it may be built with one or more certificate validation engines, for example, Windows SChannel, OpenSSL, NSS etc. For example, our demo session running on Windows shows

(src) 75 % curl::version
TclCurl Version 7.74.0 (libcurl/7.74.0 Schannel zlib/1.2.11 WinIDN)

Depending on how it was built, multiple engines may be supported and can even be configured at runtime. The key thing to remember that the options and the behavior may change based on the engine in use, build-time configuration and even environment variables at run time. Care must be taken that default behaviors match expectations.

And talking about unexpected default behaviors brings us to TclTls.

Validation with TclTls

As we saw earlier, both TWAPI and curl (at least with the Schannel engine) will fail on the connect if server certificate validation fails. Let us try the same with TclTls.

(src) 82 % set so [tls::socket expired.badssl.com 443]
sock000002E4985A7420
(src) 83 % httpreq $so expired.badssl.com 
(src) 84 % read $so
HTTP/1.1 200 OK
Server: nginx/1.10.3 (Ubuntu)
Date: Fri, 19 Feb 2021 11:04:37 GMT
Content-Type: text/html
... Additional lines omitted ...

As seen above, the connection is made successfully despite the server having an invalid certificate. And just to verify this behavior on non-Windows platforms, we can try on Ubuntu 20.04 with the system provided Tcl and TclTls.

% set so [tls::socket expired.badssl.com 443]
sock5599381b9cc0
% httpreq $so expired.badssl.com
% read $so
HTTP/1.1 200 OK
Server: nginx/1.10.3 (Ubuntu)
Date: Fri, 19 Feb 2021 11:23:53 GMT
Content-Type: text/html
... Additional lines omitted ...

We see the same behavior there as well.

The TclTls extension in fact needs the -require option to be explicitly specified as true to enable certificate validation.

(src) 105 % set so [tls::socket -require 1 expired.badssl.com 443]
(src) 108 % httpreq $so expired.badssl.com
error flushing "sock000002E495AABA20": software caused connection abort

The error message is a little misleading. To get a clearer indication, we can use tls::handshake that does an explicit negotiation.

(src) 109 % set so [tls::socket -require 1 expired.badssl.com 443]
sock000002E4985A8B20
(src) 110 % tls::handshake $so
handshake failed: certificate verify failed

The error message is now clearer.

This silent default of not checking certificates does not seem appropriate to me from a security function point of view. The user (programmer) in all likelihood expects the connection to have been secured when in fact it isn't. See The Most Dangerous Code in the World [PDF] for more on this.

Be as it may, once we know about the default behavior, we can make sure to always specify the -require option to force certificate validation.

Unfortunately, this has another effect as we see when we try to retrieve a URL with a valid certificate:

(src) 111 % set so [tls::socket -require 1 www.example.com 443]
sock000002E4985D24D0
(src) 112 % tls::handshake $so
handshake failed: certificate verify failed

Now we cannot retrieve legitimate sites! The issue here is that OpenSSL libraries do not know the location of the trusted root certificates on the system. This is understandable on Windows as OpenSSL does not look into the Windows certificate store. But we see the same on Ubuntu even using the system supplied packages which is a little surprising.

To fix this, we have to supply the path to the certificate store using the -cadir option. On Ubuntu:

% set so [tls::socket -require 1 -cadir /usr/share/ca-certificates/mozilla www.example.com 443]
sock5599381a5370
% tls::handshake $so
1

or even using an alternative path to a certificate store,

% set so [tls::socket -require 1 -cadir /etc/ssl/certs www.example.com 443]
sock55993819ec60
% tls::handshake $so
1

Now the handshake completes successfully. Just to ensure bad certificates are still detected,

% set so [tls::socket -require 1 -cadir /usr/share/ca-certificates/mozilla expired.badssl.com 443]
sock561ed2479c70
% tls::handshake $so
handshake failed: certificate verify failed

This now all works for Ubuntu but how about on Windows which has no OpenSSL certificate store?

There are two options here:

TclTls also has a callback option for certificate validation. You can use this to extract the server certficate and then use TWAPI's certificate services as described earlier to validate against the Windows certificate store. This is slightly involved as the certificate formats are different. But if you are dropping down to TWAPI for validation anyways, why not use TWAPI's tls_socket command?
The other option is to download a trusted store archive that can then be passed in to TclTls (really OpenSSL). One such is the cacert.pem file downloadable from the curl web site. This file which is updated regularly contains the trusted root store maintained by Mozilla as a single PEM file. This can be downloaded and passed to TclTls with the -cafile option (as opposed to -cadir which expects a directory). Of course, you have to trust the site from where you downloaded the store just as you trust Microsoft with the Windows store.

Illustrating this second option, with the store being downloaded to the `D:/temp' directory,

(src) 113 % 
(src) 113 % set so [tls::socket -require 1 -cafile d:/temp/cacert.pem www.example.com 443]
sock000002E498731F50
(src) 114 % tls::handshake $so
1

Note that this option has its own issues. The downloaded certificate store has to be regularly updated to account for revocations, new and outdated trust certificates etc. On Linux/Unix one may expect the system updates to do the needful. On Windows it is up to the application to do so. TWAPI and TclCurl (with Schannel) do not have this issue because Windows update will also update the Windows certificate store.

Validation summary

So what is the bottom line lesson for TLS validation? Verify for your specific environment (platform, package used, build configuration, phase of the moon...) that certificate validation works correctly not just for legitimate sites (hard to miss because users will complain) but also for illegitimate ones (easy to miss because no one might notice).

Converting a channel to use TLS

Neither TclTLS nor TWAPI is restricts use of TLS to TCP socket at connection set up time.

TLS may be layered over any channel type, for example named pipes on Windows.
Any channel that is already in use may be switched to use TLS at any time. For example, a SMTP client can send a STARTTLS command to a server to upgrade an insecure connection to a secure one over TLS.

The TclTls tls::import command and the TWAPI twapi::starttls command are used for this purpose.

(src) 116 % set so [socket www.example.com 443]
sock000002E495AED710
(src) 117 % set tls_so [twapi::starttls $so -peersubject www.example.com]
rc12

The channel returned by twapi::starttls is then used for communication. The original socket should not be directly used further.

The TclTls tls::import command works similarly and not illustrated here.

Protocol options

All three packages support options that control various attributes of the TLS protocol used for a connection, such as protocol version, ciphers, etc. See the respective documentation pages for more on these.

Choosing a package

Given the different packages, which package should you choose for you application?

TclCurl is cross-platform and has very wide application protocol support. The other two are strictly transport level and any application protocols have to layered on top, for example using the http package in the Tcl core. On the other hand, TclCurl does not provide access to a raw transport channel (socket).
TclTls is also cross-platform and provides a raw TLS socket over which any application protocol can be layered. You need to take care to mandate certificate validation and explicitly provide a path to the certificate store which may be located in different places on different systems. Moreover, on systems like Windows where this private certificate store is not automatically updated, you need to provide a means for update.
TWAPI is not cross-platform but provides similar capabilities to TclTls on Windows while being much simpler to maintain as it makes use of the Windows certificate store and automatic updates.

Note that the API's are similar enough that you can support both TclTls and TWAPI on a platform-specific basis.