Memory leak in FR 3.0.19 ?

classic Classic list List threaded Threaded
6 messages Options
| Threaded
Open this post in threaded view
|

Memory leak in FR 3.0.19 ?

Arnaud LAURIOU
Hi,

We are suspecting memory leak in FR 3.0.19 : process memory status show
VmData always keep
increasing leading to VmSwap use after a few days which also keep
increasing.

I have 2 questions :
- What are the compile options to use valgrind ? only flag
--enable-developper ?||
- In the changelog for version 3.0.20 : some memories problems seems to
be solved, do you
know when this version will be released ? (if the release date is close,
we can wait to test this
release rather than 3.0.19)

Regards,

Arnaud Lauriou
-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
| Threaded
Open this post in threaded view
|

Re: Memory leak in FR 3.0.19 ?

Matthew Newton-3
On Fri, 2019-10-11 at 15:15 +0200, Arnaud LAURIOU wrote:
> know when this version will be released ? (if the release date is
> close, we can wait to test this release rather than 3.0.19)

If you try the v3.0.x branch from GitHub, we could see if there's a
problem _before_ 3.0.20 is released...

--
Matthew


-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
| Threaded
Open this post in threaded view
|

Re: Memory leak in FR 3.0.19 ?

Alan DeKok-2
In reply to this post by Arnaud LAURIOU
On Oct 11, 2019, at 9:15 AM, Arnaud LAURIOU <[hidden email]> wrote:
>
> We are suspecting memory leak in FR 3.0.19 : process memory status show VmData always keep
> increasing leading to VmSwap use after a few days which also keep increasing.
>
> I have 2 questions :
> - What are the compile options to use valgrind ? only flag --enable-developper ?||

  valgrind is a run-time checker, and doesn't need any compile time changes.  You just run:

        valgrind --tool=memcheck --leak-check=full radiusd -f ...

  Note that you will need the "-f".

> - In the changelog for version 3.0.20 : some memories problems seems to be solved, do you
> know when this version will be released ? (if the release date is close, we can wait to test this
> release rather than 3.0.19)

  As Matthew says, you can test it now.

  Alan DeKok.


-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
| Threaded
Open this post in threaded view
|

Re: Memory leak in FR 3.0.19 ?

Arnaud LAURIOU

On 10/11/19 3:33 PM, Alan DeKok wrote:

> On Oct 11, 2019, at 9:15 AM, Arnaud LAURIOU<[hidden email]>  wrote:
>> We are suspecting memory leak in FR 3.0.19 : process memory status show VmData always keep
>> increasing leading to VmSwap use after a few days which also keep increasing.
>>
>> I have 2 questions :
>> - What are the compile options to use valgrind ? only flag --enable-developper ?||
>    valgrind is a run-time checker, and doesn't need any compile time changes.  You just run:
>
> valgrind --tool=memcheck --leak-check=full radiusd -f ...
>
>    Note that you will need the "-f".
>
>> - In the changelog for version 3.0.20 : some memories problems seems to be solved, do you
>> know when this version will be released ? (if the release date is close, we can wait to test this
>> release rather than 3.0.19)
>    As Matthew says, you can test it now.
Thank's for your answers, here are some logs after testing FR 3.0.20
with about a
hundred authentications (let me know if you need more specific information
from valgrind, I don't know it well) :

==27160== LEAK SUMMARY:
==27160==    definitely lost: 9,390 bytes in 50 blocks
==27160==    indirectly lost: 308,580 bytes in 2,163 blocks
==27160==      possibly lost: 78,682,289 bytes in 13,374 blocks
==27160==    still reachable: 1,779,086 bytes in 31,899 blocks
==27160==                       of which reachable via heuristic:
==27160==                         length64           : 53,360 bytes in
356 blocks
==27160==                         newarray           : 1,936 bytes in 22
blocks
==27160==         suppressed: 0 bytes in 0 blocks
==27160== Reachable blocks (those to which a pointer was found) are not
shown.

blocks 'definitely lost' :

==27160== 32,936 (136 direct, 32,800 indirect) bytes in 1 blocks are
definitely lost in loss record 1,499 of 1,529
==27160==    at 0x4C31B25: calloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==27160==    by 0x840B31E: gnutls_x509_crt_init (in
/usr/lib/x86_64-linux-gnu/libgnutls.so.30.14.10)
==27160==    by 0x84102A8: gnutls_x509_crt_list_import (in
/usr/lib/x86_64-linux-gnu/libgnutls.so.30.14.10)
==27160==    by 0x78C5A93: ??? (in
/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.10.8)
==27160==    by 0x78C342F: ??? (in
/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.10.8)
==27160==    by 0x78B968C: ldap_set_option (in
/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.10.8)
==27160==    by 0x7680B53: mod_conn_create (ldap.c:1532)
==27160==    by 0x137CDC: fr_connection_spawn (connection.c:438)
==27160==    by 0x13859F: fr_connection_get_internal (connection.c:858)
==27160==    by 0x767B528: mod_authorize (rlm_ldap.c:1465)
==27160==    by 0x131762: call_modsingle (modcall.c:304)
==27160==    by 0x131762: modcall_recurse (modcall.c:580)
==27160==    by 0x1310A2: modcall_child (modcall.c:410)


==27160== 34,794 (4,320 direct, 30,474 indirect) bytes in 9 blocks are
definitely lost in loss record 1,502 of 1,529
==27160==    at 0x4C31B25: calloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==27160==    by 0x83FFF47: gnutls_x509_privkey_init (in
/usr/lib/x86_64-linux-gnu/libgnutls.so.30.14.10)
==27160==    by 0x78C5A12: ??? (in
/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.10.8)
==27160==    by 0x78C342F: ??? (in
/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.10.8)
==27160==    by 0x78B968C: ldap_set_option (in
/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.10.8)
==27160==    by 0x7680B53: mod_conn_create (ldap.c:1532)
==27160==    by 0x137CDC: fr_connection_spawn (connection.c:438)
==27160==    by 0x138228: fr_connection_pool_check (connection.c:728)
==27160==    by 0x767B677: mod_authorize (rlm_ldap.c:1673)
==27160==    by 0x131762: call_modsingle (modcall.c:304)
==27160==    by 0x131762: modcall_recurse (modcall.c:580)
==27160==    by 0x1310A2: modcall_child (modcall.c:410)
==27160==    by 0x13124B: modcall_recurse (modcall.c:791)


==27160== 61,216 (304 direct, 60,912 indirect) bytes in 2 blocks are
definitely lost in loss record 1,515 of 1,529
==27160==    at 0x4C31B25: calloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==27160==    by 0x9F4305B: ??? (in
/usr/lib/x86_64-linux-gnu/libtasn1.so.6.5.5)
==27160==    by 0x9F4326C: asn1_create_element (in
/usr/lib/x86_64-linux-gnu/libtasn1.so.6.5.5)
==27160==    by 0x840B343: gnutls_x509_crt_init (in
/usr/lib/x86_64-linux-gnu/libgnutls.so.30.14.10)
==27160==    by 0x84102A8: gnutls_x509_crt_list_import (in
/usr/lib/x86_64-linux-gnu/libgnutls.so.30.14.10)
==27160==    by 0x78C5A93: ??? (in
/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.10.8)
==27160==    by 0x78C342F: ??? (in
/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.10.8)
==27160==    by 0x78B968C: ldap_set_option (in
/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.10.8)
==27160==    by 0x7680B53: mod_conn_create (ldap.c:1532)
==27160==    by 0x137CDC: fr_connection_spawn (connection.c:438)
==27160==    by 0x138228: fr_connection_pool_check (connection.c:728)
==27160==    by 0x767B677: mod_authorize (rlm_ldap.c:1673)


==27160== 143,384 (1,224 direct, 142,160 indirect) bytes in 9 blocks are
definitely lost in loss record 1,520 of 1,529
==27160==    at 0x4C31B25: calloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==27160==    by 0x840B31E: gnutls_x509_crt_init (in
/usr/lib/x86_64-linux-gnu/libgnutls.so.30.14.10)
==27160==    by 0x84102A8: gnutls_x509_crt_list_import (in
/usr/lib/x86_64-linux-gnu/libgnutls.so.30.14.10)
==27160==    by 0x78C5A93: ??? (in
/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.10.8)
==27160==    by 0x78C342F: ??? (in
/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.10.8)
==27160==    by 0x78B968C: ldap_set_option (in
/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.10.8)
==27160==    by 0x7680B53: mod_conn_create (ldap.c:1532)
==27160==    by 0x137CDC: fr_connection_spawn (connection.c:438)
==27160==    by 0x138228: fr_connection_pool_check (connection.c:728)
==27160==    by 0x767B677: mod_authorize (rlm_ldap.c:1673)
==27160==    by 0x131762: call_modsingle (modcall.c:304)
==27160==    by 0x131762: modcall_recurse (modcall.c:580)
==27160==    by 0x1310A2: modcall_child (modcall.c:410)


Issue with the ldap module ?

Regards,

Arnaud Lauriou
-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
| Threaded
Open this post in threaded view
|

Re: Memory leak in FR 3.0.19 ?

Alan DeKok-2
On Oct 18, 2019, at 8:26 AM, Arnaud LAURIOU <[hidden email]> wrote
> Thank's for your answers, here are some logs after testing FR 3.0.20 with about a
> hundred authentications (let me know if you need more specific information
> from valgrind, I don't know it well) :

  That's OK.

> blocks 'definitely lost' :
>
> ==27160== 32,936 (136 direct, 32,800 indirect) bytes in 1 blocks are definitely lost in loss record 1,499 of 1,529
> ==27160==    at 0x4C31B25: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
> ==27160==    by 0x840B31E: gnutls_x509_crt_init (in /usr/lib/x86_64-linux-gnu/libgnutls.so.30.14.10)

  And that's a problem.

  DON'T use GnuTLS with FreeRADIUS.  It's wrong and broken.  For details, see:

https://networkradius.com/freeradius-packages/

  Note: CentOS and RedHat link their LDAP libraries against NSS. FreeRADIUS uses OpenSSL. NSS and OpenSSL cannot be used in the same application, as they will cause it to crash. FreeRADIUS therefore must use libldap from the LDAP Toolbox Project, which uses OpenSSL.

> Issue with the ldap module ?

  Issues with GNU TLS.  Don't use it.

  Install libldap from the LTB project, and install the FreeRADIUS packages from the Network RADIUS web site.  It should work.

  If you still see memory leaks, then we can fix them. But there is absolutely no way that we can fix GNU TLS nonsense.  Especially when the leak is buried inside of libldap.  We didn't write libldap, and we know nothing about it.  The only thing we can say is "use a version of libldap that isn't broken".

  Alan DeKok.


-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
| Threaded
Open this post in threaded view
|

Re: Memory leak in FR 3.0.19 ?

Arnaud LAURIOU

On 10/18/19 3:04 PM, Alan DeKok wrote:

>> blocks 'definitely lost' :
>>
>> ==27160== 32,936 (136 direct, 32,800 indirect) bytes in 1 blocks are definitely lost in loss record 1,499 of 1,529
>> ==27160==    at 0x4C31B25: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
>> ==27160==    by 0x840B31E: gnutls_x509_crt_init (in /usr/lib/x86_64-linux-gnu/libgnutls.so.30.14.10)
>    And that's a problem.
>
>    DON'T use GnuTLS with FreeRADIUS.  It's wrong and broken.  For details, see:
>
> https://networkradius.com/freeradius-packages/
>
>    Note: CentOS and RedHat link their LDAP libraries against NSS. FreeRADIUS uses OpenSSL. NSS and OpenSSL cannot be used in the same application, as they will cause it to crash. FreeRADIUS therefore must use libldap from the LDAP Toolbox Project, which uses OpenSSL.
>
>> Issue with the ldap module ?
>    Issues with GNU TLS.  Don't use it.
>
>    Install libldap from the LTB project, and install the FreeRADIUS packages from the Network RADIUS web site.  It should work.
>
>    If you still see memory leaks, then we can fix them. But there is absolutely no way that we can fix GNU TLS nonsense.  Especially when the leak is buried inside of libldap.  We didn't write libldap, and we know nothing about it.  The only thing we can say is "use a version of libldap that isn't broken".
>
Hi,

This email just to confirm that we don't have any more memory leaks with
FR 3.0 using the LTB libraries.

Thanks,

Arnaud Lauriou

-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html