Handshake fails with Internal error in memory allocation

[Nikos Mavrogiannopoulos]

Andreas Metzler wrote:
> On 2008-04-29 Simon Josefsson <simon at josefsson.org> wrote:
>> This error has come up lately, see:
> 
>> http://bugs.debian.org/466477
>> http://bugs.debian.org/478191
> 
>> The cause seems clear, the server sends a huge list of CA certs and
>> GnuTLS runs into some fixed size buffer or something.  This reproduces
>> it:
> 
>> gnutls-cli -p 25 -s mail3.mclemente.net
>> ehlo foo
>> starttls
>> ^D
>> Nikos, do you have any idea?  I could look at it, but have little time
>> right now.
> Hello,
> 
> isn't it a bug that gnutls *sends* this huge list of certificates in
> the first place? (Noted by Florian Weimer)?

Not really. In the TLS handshake, when the server requests a certificate
from the client, it has to indicate which CAs he trusts so the client
can send an appropriate certificate.

> I think this is rather strange:
> 
> Start with this setup:
>   - Server is using a self signed certificate and key.
>   - Client is not using any certificate at all.
> 
> This works ...
> *server* gnutls-serv --port 666 --x509certfile /etc/exim4/exim.crt \
>     --x509keyfile /etc/exim4/exim.key
> *client* gnutls-cli localhost -p 666

This is a case where the server (since no --x509cafile is specified)
does not advertize the CA certificates that he trusts.

> ... but this suddenly doesn't (with
> the old #define MAX_HANDSHAKE_PACKET_SIZE 16*1024):
> *server*  gnutls-serv --port 666 --x509certfile /etc/exim4/exim.crt \
>     --x509keyfile /etc/exim4/exim.key \
>     --x509cafile /etc/ssl/certs/ca-certificates.crt
> *client* gnutls-cli localhost -p 666

In this case the server trusts as many certificates as they are in
ca-certificates.crt. Thus the advertisement is quite long.

> I do not understand why specifying a list of irrelevant trusted CAs
> changes the the TLS dialogue at all. Afaict this is not the case for
> openssl, this won't break gnutls:
>  openssl s_server -accept 666 -cert /etc/exim4/exim.crt
>    -key  /etc/exim4/exim.key -CAfile /etc/ssl/certs/ca-certificates.crt 

gnutls has a maximum size for handshake message, which overflowed with
this long CA list (could be unlimited but is set to a specific value to
avoid possible denial of service). As far as I know this limit was
increased in the latest gnutls.

A server that trusts quite many CAs might use:
gnutls_certificate_free_ca_names(), to avoid sending all of the to the
client. In that case it might be desirable to do so, since the
information that they provide to the client is not really much.

regards,
Nikos