Why am I getting a segfault when trying to use Dr Brian Gladman's AES-GCM?

I am trying to use Dr Brian Gladman's famous AES encryption routines, in this case the Galois Counter Mode version, and I'm getting a segfault and can't find the cause.

I've written a little demo program that shows what I'm trying to do and in fact it appears to work, but at the end it segfaults.

I am a little rusty in C, so the error may be obvious to wiser heads.

Here's my program, which I hope is as short and readable as possible:

/* This program tries to be a demo for Dr Brian Gladman's AES-GCM encryption code.
   The plan is to use AES-GCM to encrypt a short message, and provide an authentication key.
   And then to decrypt the ciphertext, verifying the authentication tag on the way.*/

#include<stdio.h>
#include<assert.h>

#include "aes-modes/gcm.h"

/*note that uint_8t is Dr Gladman's type, not to be confused with uint8_t from C99*/

void print_uint_8t_array(uint_8t *a, int len, char* name)
{
  int i;
  printf("%-10s:", name);

  for(i=0; i< len; i++){
    if(a[i]==0x00) printf("-");
    else if(a[i]>=0x20 && a[i]<=0x7e) printf("%c", a[i]); 
    else printf("~");
  }
  printf("\n");
}

#define p8(name) (print_uint_8t_array((name), (sizeof(name)), (#name)))

/*Dr Gladman's functions return 0,1 or -1, to be interpreted so:*/
void interpret_retval(AES_RETURN retval){
  switch(retval){
  case      0: 
    printf("RETURN_GOOD\n");
    break;
  case 1:
    printf("RETURN_WARN\n");
    break;
  case -1:
    printf("RETURN_ERROR\n");
    break;
  default: printf("Unknown return value");
    assert(0);
    break;
  }
}

int main(void){
  printf("GCM-AES encryption/decryption/authentication example\n");

  aes_init(); 
  /*I think that this is only needed if you've compiled the libraries to make up
    their tables at runtime. If the tables are compiled in, then it's
    unnecessary but probably doesn't do any harm.*/

  uint_8t key[32]="01234567890123456789012345678901"; 
  uint_8t iv[32]="01234567890123456789012345678901";

  uint_8t header[]="Here is a header, which is not to be encrypted, but which is nevertheless to be proved to have been placed in the message by someone who knows the key.";
  uint_8t message[]="Here is some text which is to be protected from the prying eyes of those who do not know the key, whilst at the same time it is to carry along with it an unencrypted header, and a with them both a tag proving beyond reasonable doubt that the two were packaged together and run through the aes-gcm algorithm by someone who knew the key."; 
  uint_8t tag[32]="01234567890123456789012345678901";

  void printtexts(void)
  {
    printf("-----------------------------------\n");
    p8(key);
    p8(iv);
    printf("-----------------------------------\n");
    p8(header);
    p8(message);
    p8(tag);
    printf("-----------------------------------\n");
  }

  printtexts();

  {
    gcm_ctx ecx[1];
    printf("encrypting....\n");
    gcm_init_and_key(key, sizeof(key), ecx);
    interpret_retval(gcm_encrypt_message(iv, sizeof(iv), header, sizeof(header), message, sizeof(message), tag, sizeof(tag), ecx));
    gcm_end(ecx);
    printf("done\n");
  }

  printtexts();

  {
    gcm_ctx dcx[1];

    printf("decrypting....\n");
    gcm_init_and_key(key, sizeof(key), dcx);
    interpret_retval(gcm_decrypt_message(iv, sizeof(iv), header, sizeof(header), message, sizeof(message), tag, sizeof(tag), dcx));
    gcm_end(dcx);
    printf("done\n");
  }

  printtexts();

  return 0;

}

And here's the result. The encryption/decryption and tagging all appear to work, but then there's a bad return code and a segfault.

jla@jaspden-des开发者_高级运维ktop$ ./test3
GCM-AES encryption/decryption/authentication example
-----------------------------------
key       :01234567890123456789012345678901
iv        :01234567890123456789012345678901
-----------------------------------
header    :Here is a header, which is not to be encrypted, but which is nevertheless to be proved to have been placed in the message by someone who knows the key.-
message   :Here is some text which is to be protected from the prying eyes of those who do not know the key, whilst at the same time it is to carry along with it an unencrypted header, and a with them both a tag proving beyond reasonable doubt that the two were packaged together and run through the aes-gcm algorithm by someone who knew the key.-
tag       :01234567890123456789012345678901
-----------------------------------
encrypting....
RETURN_GOOD
done
-----------------------------------
key       :01234567890123456789012345678901
iv        :01234567890123456789012345678901
-----------------------------------
header    :Here is a header, which is not to be encrypted, but which is nevertheless to be proved to have been placed in the message by someone who knows the key.-
message   :~(}Hr~|o~~:~D~~~~3~~T~C~C(l~~~'~~~UC~D~~~~&O~ ~L~-~~~E&~~~~!~~~~~K~~~~M<|l%<ho~"~[A~0~~O~3T~%8K~~~L~~~~~~~~~7~~~~~~~~e~~~~;~392~8<~~ ~~v,E~~~~~~~~~~~W~yd~~C$H~~~*r~~~_~~~~O~u~h\9s~~`%~~~~~&~~~~~~.~~Q~~~Y~Ix~A~~~a5~~~~|~~9J~~~~~~~~ ~~ZOiX~~~~~~f~~`e~~~Ju~Z~~~X~g~OX~C~~~~~~~~~(~~gyT~~x~L~4>~Z~~ge~~~08~;~@~s~3~~WA~~~~Z-~~~~/~~~~~~Tj~bL~~
tag       :|~~,~2~~Ec~~~A~~~~~_j{~~~~`~~~u~
-----------------------------------
decrypting....
RETURN_ERROR
done
-----------------------------------
key       :01234567890123456789012345678901
iv        :01234567890123456789012345678901
-----------------------------------
header    :Here is a header, which is not to be encrypted, but which is nevertheless to be proved to have been placed in the message by someone who knows the key.-
message   :Here is some text which is to be protected from the prying eyes of those who do not know the key, whilst at the same time it is to carry along with it an unencrypted header, and a with them both a tag proving beyond reasonable doubt that the two were packaged together and run through the aes-gcm algorithm by someone who knew the key.-
tag       :|~~,~2~~Ec~~~A~~~~~_j{~~~~`~~~u~
-----------------------------------
Segmentation fault (core dumped)

Self-answering, in case anyone else is interested in using these routines.

Apparently the maximum tag size for AES-GSM is 16 bytes. Change the tag-length to 16 or less (I have it as 32 here), and everything works fine.

This limit appears to be hard-coded into gcm_decrypt_message as BLOCK_SIZE. Changing that to 32 also seems to work, but the Lord alone knows what that does to the cryptographic properties of the algorithm.

On a debugging note, gcc has an option -fstack-protector-all which can catch libraries in the act of smashing the return stack, which was what was happening here.

As far as I can tell, the segfault is caused when main tries to return to a completely fictional address which has been written onto the stack by the library.