Cryptography on Windows Part 5 - Symmetric cryptography II

Published , updated

Having gone through several introductory posts covering background material such as CSP's, contexts, key containers and methods of key generation, we are now ready to delve into the actual cryptographic operations that use symmetric algorithms in detail.

Our usual reminder — the code snippets here assume you have loaded the TWAPI package and set up the namespace path to include its namespace. Also the convenience procedure hex is defined to dump binary data in hexadecimal format. 

package require twapi
namespace path twapi
proc hex bin {  binary encode hex $bin }

Encryption using high level wrappers

For starters, we will ignore all this background material from prior posts and instead look at the use of wrappers that TWAPI provides that encapsulate the low level operations into simple commands for encryption and decryption. Note however, that this simplicity comes at a cost in efficiency as the underlying operations of retrieving a context, initializing keys etc. are needlessly repeated on every call. Thus these commands are recommended only in cases where cryptographic operations are only occasionally required. Moreover, these commands only encapsulate commonly-used cryptographic algorithms. Finally, these wrapper commands are only supported in TWAPI 4.3 or later. Thus for better performance, employing less common algorithms or for earlier versions of TWAPI, you will have to resort to the low level operations described in later sections.

In their simplest form, the wrappers take the input data and a key and perform the specified cryptographic operation and return the result. 

ALGOCMD OP INPUT KEY

The cryptographic algorithm in use is identified by the command ALGOCMD while OP is encrypt or decrypt specifying the operation to be performed. Correspondingly, INPUT is the plaintext for encryption and ciphertext for decryption.

Thus the following encrypts the string Hello world! using the DES algorithm and a (presumably previously agreed) super-secret shared key consisting of all zeroes. 

% set key [conceal "\0\0\0\0\0\0\0\0"]
% set enc [des encrypt "Hello world!" $key] ; hex $enc
b46c3025a77dffad2713ab7bfa3a6be6

As I mentioned in earlier posts, most cryptography related commands in TWAPI expect keys themselves to be in an encrypted form in memory. The same is true here as well and hence we pass in our secret key argument after converting it to such a form with the conceal command. (I have been promising to discuss conceal and related commands for several posts now; I will definitely do so in the next post in this series.) And of course, although we are using a hard-coded key here for illustrative purposes, you could (and should) use any of the other key derivation methods, such as passphrase based keys, that we discussed in our previous post.

The decryption command is similar. For symmetric algorithms, the decryption key is the same as the encryption key. 

% des decrypt $enc $key
Hello world!

Also important to note, the wrapper commands expect the input data to be a binary string. If you are encrypting a character string, it must first be converted to a binary string using a suitable encoding. When the plaintext is pure ASCII, as above, this is not an issue but otherwise it must be transformed before passing to the encryption routines. For example, the above encryption should be written as 

% set enc [des encrypt [encoding convertto utf-8 "Hello world!"] $key]

Obviously, at the decryption end, if character data that was encoded as above, a reverse transformation is required. For example, 

% encoding convertfrom utf-8 [des decrypt $enc $key]

Analogous to the des command for the DES cipher, the 3des, aes_128, aes_192 and aes_256 commands work with the 3DES and 128-bit, 192-bit and 256-bit AES ciphers respectively. The key size must of course be appropriate for the cipher in use — 8 bytes for DES (out which 56 bits are significant), 16 bytes for AES 128 and so on. Note that although our examples above used DES, modern applications should use one of the AES ciphers as DES is considered too weak given the computational power of today's machines.

Cipher modes

As we mentioned in our introductory post, block ciphers may operate in one of several modes such as Electronic Code Book (ECB), Cipher Block Chaining (CBC) etc. These modes refer to how blocks of plaintext affect encryption of succeeding blocks. Our examples above did not specify the mode which then defaulted to CBC as that is the default assumed by the Microsoft CSP's.

Let us see the effect of CBC mode by looking at the following output. 

% hex [des encrypt 01234567abcdefgh $key]
a068dbeab73d140b0c0fd00fe63bc1b5738d5d1aff1cb396
% hex [des encrypt 00000000abcdefgh $key]
de8d55142b18deb76b89aa117a89ce426c6b7d627f3057dd

Notice how the second block of eight bytes, abcdefgh in both examples, is encrypted to different ciphertext. This is because in CBC mode (the default since we did not specify a mode) the ciphertext for a block is produced by exclusive-oring the ciphertext of the previous block with the plaintext of the current block before passing it through the encryption transform.

On the other hand, ECB mode encrypts each block independently as seen by the following output. 

% hex [des encrypt 01234567abcdefgh $key -mode ecb]
a068dbeab73d140ba844348fa6fd93607e422822773666c0
% hex [des encrypt 00000000abcdefgh $key -mode ecb ]
de8d55142b18deb7a844348fa6fd93607e422822773666c0

As you see above, in ECB mode the last 8 bytes produce the same ciphertext. This property of ECB mode, producing the same ciphertext for a given plaintext, is a serious weakness and in general ECB mode should never be used in an application.

In addition to CBC, there are other modes of operation that achieve a similar effect in producing different output for a given plaintext block. However, as of the time of writing, Microsoft-supplied CSP's only support ECB, CBC and CFB modes. ECB is insecure and CFB padding conventions vary between implementations (e.g. OpenSSL differs from CryptoAPI). Thus in practice CBC seems to be the only choice when using Microsoft CSP's.

Initialization vectors

There is one question that may have popped into your mind from the above regarding CBC mode. Since the encryption output for a plaintext block depends on the previous one, how does the very first block get encrypted? This is where the initialization vector comes in. This is some random "dummy" data that is used in producing the ciphertext for the first real data block in place of the preceding ciphertext block that is used for subsequent blocks. This must necessarily be of the same size as the native block size for the cipher.

When unspecified, Microsoft CSP's will default to using a sequence of bytes consisting of all zeroes as the initialization vector. This does mean that that message containing the same plaintext in the first block will be encrypted to have the same ciphertext in the first block. Since this is not desirable, it is crucial to not to use the same initialization vector with the same key. We can achieve this by specifying a random sequence of bytes of the block size as the initialization vector.

In TWAPI the random_bytes command produces a cryptographically sound stream of random bytes. For further convenience, the des iv, aes_128 iv etc. commands will produce the right number of bytes for the corresponding algorithm. These can be passed to the appropriate encryption command with the -iv option. For example, 

% set iv [des iv]
% set enc [des encrypt "Hello world!" $key -mode cbc -iv $iv]

For decryption to work, this IV has to be sent to the receiver as well. For example it can simply be prepended to the encrypted ciphertext and stripped off by the receiver. For example, assuming a transmit procedure that sends a message, the following would send the IV and ciphertext to the receiver. 

set msg "$iv$enc"
transmit $msg

The receiver can strip off the IV (knowing it is 8 bytes for DES) and pass it to the decrypt command. 

% des decrypt [string range $msg 8 end] $key -mode cbc -iv [string range $msg 0 7]
Hello world!

Note that this IV need not be kept secret but it is crucial that a new random, non-predictable IV is generated for every message that uses that same key.

NOTE: Because the default mode and IV may depend on the specific CSP we have chosen, it is always recommended to explicitly specify the -mode and -iv options to the commands. Needless to say, the decrypting side needs to specify the same values for those options.

Encryption using low level commands

In the previous section, I described the high level commands, such as des, that encrypt and decrypt messages in a single call. Under the hood, these commands

  • create up an appropriate cryptographic context

  • set up and initialize key structure including encryption modes and initialization vectors

  • invoke the actual encryption commands

  • release all allocated structures

Encapsulation of all these operations provide ease-of-use and convenience but come with some drawbacks.

  • The repeated allocation and release of contexts and key structures has a negative performance impact.

  • There is no way to operate in streaming mode, i.e. the entire message must be assembled before being passed to the encryption or decryption.

  • There is no mechanism to select a different CSP, for example a hardware-based accelerator, or one that supports additional cipher modes.

The low level commands that I now describe provide more control and flexibility that do not have these drawbacks at the cost of a little more complexity.

Creating a cryptographic context

The first step is to create a cryptographic context that supports the algorithm of choice. 

% set sender_hcrypt [crypt_acquire -csptype prov_rsa_aes]
1685163529632 HCRYPTPROV

I already described this process in a prior post so you can refer there for details.

Establishing a key

The next step is to establish a key that will be used for the cryptographic operations. In our previous post we saw a couple of methods by which symmetric keys can be generated. Here I will instead assume we have a raw key obtained by some out of band means given to us as in the examples above. 

% set raw_key [string repeat \0 16]

We are again using our magical all-zeroes key except that because we plan on using the AES 128 cipher, it is a longer key than the one we used for DES.

To use a raw key for cryptographic operations, it has to be first imported into the cryptographic context with the crypt_import_key command. To do this, we have to first transform it into a key blob structure. 

% set keyblob [capi_keyblob_plaintext aes_128 $raw_key]

The capi_keyblob_plaintext command was used because the raw key itself was in plaintext form, i.e. not protected in memory. If it was in concealed form, the capi_keyblob_concealed command would have to be used instead.

We can now import the raw key into the cryptographic context. This will return a key handle that we can then use for cryptographic operations. 

% set sender_hkey [crypt_import_key $sender_hcrypt $keyblob]
1685162301120 HCRYPTKEY

This key handle will be used to drive our cryptographic operations.

Setting cipher modes

The cipher mode to be used for encryption defaults to CBC. We can verify this with capi_key_mode. 

% capi_key_mode $sender_hkey
cbc

The command can also be used to change the cipher mode. Since CBC is our choice, we do not need to actually set it but just for demonstration purposes: 

% capi_key_mode $sender_hkey cbc

Encrypting messages

Encryption of messages is done with the capi_encrypt_bytes and capi_encrypt_string commands. The former expects a binary string so if you are encrypting character data, it must first be transformed into a binary string using a character encoding as described earlier. The capi_encrypt_string encapsulates this by assuming the passed message is a character string and encoding it in UTF-8 before encrypting it. Thus the following would be equivalent. 

% capi_encrypt_bytes [encoding convertto utf-8 plaintext] $sender_hkey
% capi_encrypt_string plaintext $sender_hkey

The encryption of each message is a two-step process. First the initialization vector (IV) needs to be set up and then the actual encryption operation invoked. At the risk of being repetitive, remember that for the reasons mentioned earlier in this post, a different random IV must be set for every message.

To set the IV, we must generate it (as we did for the high level wrappers) and then associate with the key with the capi_key_iv command. 

% set iv [aes_128 iv]
% capi_key_iv $sender_hkey $iv

The message can now be encrypted and as before sent to the receiver along with the (unencrypted) IV. 

% set msg [capi_encrypt_string "First message" $sender_hkey]
% transmit "$iv$msg"

Subsequent messages will follow the same two-step sequence, generating and setting a new random IV, followed by the encryption operation. Note the cryptographic context and key handle remain allocated.

Streaming operation

So far we have been encrypting the entire message in a single call. Sometimes this is not convenient as it requires the entire message to be present in memory (consider a gigabyte size file). In such cases, the capi_encrypt_bytes command can be used for streaming operation where segments of the message are encrypted in sequence without requiring the entire message to be assembled.

This is accomplished through the use of the -final option to capi_encrypt_bytes. Its value is set to 1 for the last segment (indicating end of message) of a message and 0 for all others. For example, the following is equivalent to our previous call to capi_encrypt_string except that it uses capi_encrypt_bytes to encrypt the message in streaming fashion. 

set msg [capi_encrypt_bytes "First " $sender_hkey -final 0]
append msg [capi_encrypt_bytes "mess" $sender_hkey -final 0]
append msg [capi_encrypt_bytes "age" $sender_hkey -final 1]

Cleaning up

The final step when all cryptographic operations are done is to free up allocated resources. 

capi_key_free $sender_hkey
crypt_free $sender_hcrypt

Decrypting messages

The sequence for message decryption is almost identical to that of encryption with the sole exception being the use of capi_decrypt_* commands in lieu of their capi_encrypt_* counterparts. The full sequence is shown below assuming the received message is stored in the received_message variable. 

% set receiver_hcrypt [crypt_acquire -csptype prov_rsa_aes]
% set raw_key [string repeat \0 16]
% set keyblob [capi_keyblob_plaintext aes_128 $raw_key]
% set receiver_hkey [crypt_import_key $receiver_hcrypt $keyblob]
% capi_key_mode $receiver_hkey cbc
% set iv [string range $received_message 0 15]
% capi_key_iv $receiver_hkey $iv
% set msg [string range $received_message 16 end]
% set bytes [capi_decrypt_bytes [string range $msg 0 5] $receiver_hkey -final 0]
% append bytes [capi_decrypt_bytes [string range $msg 6 end] $receiver_hkey -final 1]
First message
% encoding convertfrom utf-8 $bytes
First message

The above should need no elaboration except possibly for a couple of points.

  • The initialization vector is not generated but rather extracted from the initial part of the message.

  • The decryption is a streaming operation just to illustrate that streaming can also be applied on the decrypting end.

  • Finally, the encoding convertfrom operation is a no-op here because the text was all ASCII. Otherwise it would be mandated when messages are comprised of general Unicode text.

Parallelizing encryption with digest computation

An earlier post talked about message integrity and how it can be ensured through the use of message digests and hashes. It is often the case that secure protocols make use of both encryption and message digests — one to provide privacy and the other integrity. Both these require the algorithm to iterate over the entire message leading to the potential for performance gains if they can be done concurrently in a single pass over the data.

The capi_encrypt_* and capi_decrypt_* commands provide for this by supporting a -hhash option for passing in a hash context which will be used to compute a message digest. The following sequence illustrates this. 

% set hcrypt [crypt_acquire]
2601600777072 HCRYPTPROV
% set hhash [capi_hash_create $hcrypt sha1]
2601602534144 HCRYPTHASH
% set hkey [crypt_generate_key $hcrypt des]
2601602533360 HCRYPTKEY
% hex [capi_encrypt_string "My secret message" $hkey -hhash $hhash]
fe57765580e9d4cd42daa99f3d0e45eabe201e2782504dce
% hex [capi_hash_value $hhash]
3dab81d6c1870221eda4c12fad339c1d3c200481
% capi_key_free $hkey
% capi_hash_free $hhash
% crypt_free $hcrypt

The sequence of commands above result in encryption of the message and computation of its SHA1 digest in a single pass of over the data. (Note this is illustrative so I have not bothered with IV's and such.)

Coming up next

In our series of topics related to cryptography, we are now done with discussion of message digests and symmetric algorithms. The monster called public key cryptography remains but in an effort to procrastinate, my next post will discuss Windows Data Protection — services provided by Windows to protect data within a single system or process with some simple calls that relieve the application from the headaches and complexities of managing keys.