Memory leaks are observed for libgnutls in multi-thread mode

[Nikos Mavrogiannopoulos]

tangtong wrote:
> Hi,Nikos and Simon
> 
> To verify the issue, I have configured my server to run as signle thread mode. Under high TPS, the memory leak still happen in gnutls_handshake, which means the root-cause is not caused by multi-thread. 
> 
> By more logs and analysis, I have the following findings:
> Under high TPS, my server can't serve every session timely, which leads to the closure of the sockets by the clients for timeout reason. The write operation on the server side of the corresponding socket leads to broken pipe error. gnutls_handshake() reports GNUTLS_E_PUSH_ERROR, -53. As a result, the hand-shake stage of tls session is not finished successfully. 
> 
> After repeated testing, It is evident when aborted tls session caused by error -53 are observed, the memory leak happen. 
> 
> I have double check my codes, for these aborted session, I have called the gnutls_bye()/gnutls_deinit() function.
> 
> My assumption now is for those session which has unfinished hand-shake stage, the resourses are not released properly in gnutls_handshake() for some reason.

Could you for this (memory leak) scenario to send us debugging output of
gnutls? To do that just add a logging function such as:

static void
tls_log_func (int level, const char *str)
{
  fprintf (stderr, "|<%d>| %s", level, str);
}

and call those after initialization of gnutls.
  gnutls_global_set_log_function (tls_log_func);
  gnutls_global_set_log_level (2);


regards,
Nikos