another crash on 3.0.x

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

another crash on 3.0.x

Stefan Winter-4
Hi,

with that newest 3.0.x from today, I just got a crash (after some 15 min of normal operation, so nothing about stale EAP sessions this time).

The directory in question exists (/var/log/radius/tlscache-eduroam-users/) but possibly there's simply no stored session if it's a fresh auth.

I (again) don't have an easy way to get a full backtrace as this is happening on a live system, but maybe the below gives a hint already.

Greetings,

Stefan Winter

Tue Mar 28 09:56:08 2017 : Info: rlm_sql (sql-null-acct-erx): Opening additional connection (3), 1 of 97 pending slots used
Tue Mar 28 09:56:11 2017 : Info: (12277) eap_peap:   This means you need to read the PREVIOUS messages in the debug output
Tue Mar 28 09:56:11 2017 : Info: (12277) eap_peap:   to find out the reason why the user was rejected
Tue Mar 28 09:56:11 2017 : Info: (12277) eap_peap:   Look for "reject" or "fail".  Those earlier messages will tell you
Tue Mar 28 09:56:11 2017 : Info: (12277) eap_peap:   what went wrong, and how to fix the problem
Tue Mar 28 09:56:11 2017 : Error: Couldn't open /var/log/radius/tlscache-eduroam-users/acd33e166c8284cf667a3990565aac2efa85f6ea8c0f813fba87448829b947e5.vps for reading: No such file or directory
CAUGHT SIGNAL: Segmentation fault
Backtrace of last 29 frames:
/usr/local/freeradius/3.0.x-20170328/lib/libfreeradius-radius.so(fr_fault+0x115)[0x7f9a7c155461]
/lib64/libpthread.so.0(+0xf370)[0x7f9a7aa27370]
/lib64/libssl.so.10(+0x47c92)[0x7f9a7b929c92]
/lib64/libssl.so.10(SSL_CTX_add_session+0xe7)[0x7f9a7b92aaf7]
/lib64/libssl.so.10(ssl_get_prev_session+0x40c)[0x7f9a7b92afcc]
/lib64/libssl.so.10(ssl3_get_client_hello+0x227)[0x7f9a7b8ffbd7]
/lib64/libssl.so.10(ssl3_accept+0x95d)[0x7f9a7b90481d]
/lib64/libssl.so.10(ssl23_accept+0x1c8)[0x7f9a7b912508]
/lib64/libssl.so.10(ssl23_read+0x32)[0x7f9a7b9136a2]
/usr/local/freeradius/current/sbin/radiusd(tls_handshake_recv+0x110)[0x45058c]
/usr/local/freeradius/3.0.x-20170328/lib/libfreeradius-eap.so(+0x6c51)[0x7f9a7bf40c51]
/usr/local/freeradius/3.0.x-20170328/lib/libfreeradius-eap.so(eaptls_process+0x3aa)[0x7f9a7bf410b2]
/usr/local/freeradius/3.0.x-20170328/lib/rlm_eap_peap.so(+0x1f9c)[0x7f9a75c80f9c]
/usr/local/freeradius/3.0.x-20170328/lib/rlm_eap.so(+0x40f5)[0x7f9a76ab70f5]
/usr/local/freeradius/3.0.x-20170328/lib/rlm_eap.so(eap_method_select+0x431)[0x7f9a76ab7add]
/usr/local/freeradius/3.0.x-20170328/lib/rlm_eap.so(+0x2d01)[0x7f9a76ab5d01]
/usr/local/freeradius/current/sbin/radiusd[0x428d3c]
/usr/local/freeradius/current/sbin/radiusd[0x4293e9]
/usr/local/freeradius/current/sbin/radiusd[0x428efd]
/usr/local/freeradius/current/sbin/radiusd[0x429e1d]
/usr/local/freeradius/current/sbin/radiusd(modcall+0xa2)[0x42ab62]
/usr/local/freeradius/current/sbin/radiusd(indexed_modcall+0x363)[0x426374]
/usr/local/freeradius/current/sbin/radiusd(process_authenticate+0x22)[0x428639]
/usr/local/freeradius/current/sbin/radiusd[0x40f7ed]
/usr/local/freeradius/current/sbin/radiusd(rad_authenticate+0x51e)[0x410118]
/usr/local/freeradius/current/sbin/radiusd[0x43d74c]
/usr/local/freeradius/current/sbin/radiusd[0x439382]
/lib64/libpthread.so.0(+0x7dc5)[0x7f9a7aa1fdc5]
/lib64/libc.so.6(clone+0x6d)[0x7f9a7a51773d]
Calling: gdb -silent -x /usr/local/freeradius/config/raddb/panic.gdb /usr/local/freeradius/current/sbin/radiusd 18297 2>&1 | tee /var/log/radius/gdb-radiusd-18297.log
Tue Mar 28 09:56:14 2017 : Info: Need 1 more connections to reach 5 spares
Tue Mar 28 09:56:14 2017 : Info: rlm_sql (sql-null-acct-erx): Opening additional connection (4), 1 of 96 pending slots used
Panic action exited with 0
_EXIT(0) CALLED src/lib/debug.c[743]
Tue Mar 28 10:15:49 2017 : Error: Cannot update core dump limit: Operation not permitted
Tue Mar 28 10:15:49 2017 : Info: Core dumps are enabled
Tue Mar 28 10:15:49 2017 : Info: Debugger not attached


--
Stefan WINTER
Ingenieur de Recherche
Fondation RESTENA - Réseau Téléinformatique de l'Education Nationale et de la Recherche
2, avenue de l'Université
L-4365 Esch-sur-Alzette

Tel: +352 424409 1
Fax: +352 422473

PGP key updated to 4096 Bit RSA - I will encrypt all mails if the recipient's key is known to me

http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xC0DE6A358A39DC66

-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: another crash on 3.0.x

Alan DeKok-2
On Mar 28, 2017, at 4:22 AM, Stefan Winter <[hidden email]> wrote:
> with that newest 3.0.x from today, I just got a crash (after some 15 min of normal operation, so nothing about stale EAP sessions this time).
>
> The directory in question exists (/var/log/radius/tlscache-eduroam-users/) but possibly there's simply no stored session if it's a fresh auth.
>
> I (again) don't have an easy way to get a full backtrace as this is happening on a live system, but maybe the below gives a hint already.

  That's a lot more complicated, because the crash is inside the OpenSSL functions.  i.e. not in anything we control.

  Which means that the cause is the SSL data structures being broken.  No idea how that happens.

  Alan DeKok.


-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Reply | Threaded
Open this post in threaded view
|

Re: another crash on 3.0.x

Stefan Winter-4
In reply to this post by Stefan Winter-4
Hi,

I just tried current 3.0.x and have more than an hour of runtime without
this issue re-occuring.

So, whatever it was, looks like it's fixed. I'll keep 3.0.x 3047fc6
running in production under close supervision. If it explodes, I'll sure
let the list know.

Greetings,

Stefan Winter

Am 28.03.2017 um 10:22 schrieb Stefan Winter:

> Hi,
>
> with that newest 3.0.x from today, I just got a crash (after some 15 min of normal operation, so nothing about stale EAP sessions this time).
>
> The directory in question exists (/var/log/radius/tlscache-eduroam-users/) but possibly there's simply no stored session if it's a fresh auth.
>
> I (again) don't have an easy way to get a full backtrace as this is happening on a live system, but maybe the below gives a hint already.
>
> Greetings,
>
> Stefan Winter
>
> Tue Mar 28 09:56:08 2017 : Info: rlm_sql (sql-null-acct-erx): Opening additional connection (3), 1 of 97 pending slots used
> Tue Mar 28 09:56:11 2017 : Info: (12277) eap_peap:   This means you need to read the PREVIOUS messages in the debug output
> Tue Mar 28 09:56:11 2017 : Info: (12277) eap_peap:   to find out the reason why the user was rejected
> Tue Mar 28 09:56:11 2017 : Info: (12277) eap_peap:   Look for "reject" or "fail".  Those earlier messages will tell you
> Tue Mar 28 09:56:11 2017 : Info: (12277) eap_peap:   what went wrong, and how to fix the problem
> Tue Mar 28 09:56:11 2017 : Error: Couldn't open /var/log/radius/tlscache-eduroam-users/acd33e166c8284cf667a3990565aac2efa85f6ea8c0f813fba87448829b947e5.vps for reading: No such file or directory
> CAUGHT SIGNAL: Segmentation fault
> Backtrace of last 29 frames:
> /usr/local/freeradius/3.0.x-20170328/lib/libfreeradius-radius.so(fr_fault+0x115)[0x7f9a7c155461]
> /lib64/libpthread.so.0(+0xf370)[0x7f9a7aa27370]
> /lib64/libssl.so.10(+0x47c92)[0x7f9a7b929c92]
> /lib64/libssl.so.10(SSL_CTX_add_session+0xe7)[0x7f9a7b92aaf7]
> /lib64/libssl.so.10(ssl_get_prev_session+0x40c)[0x7f9a7b92afcc]
> /lib64/libssl.so.10(ssl3_get_client_hello+0x227)[0x7f9a7b8ffbd7]
> /lib64/libssl.so.10(ssl3_accept+0x95d)[0x7f9a7b90481d]
> /lib64/libssl.so.10(ssl23_accept+0x1c8)[0x7f9a7b912508]
> /lib64/libssl.so.10(ssl23_read+0x32)[0x7f9a7b9136a2]
> /usr/local/freeradius/current/sbin/radiusd(tls_handshake_recv+0x110)[0x45058c]
> /usr/local/freeradius/3.0.x-20170328/lib/libfreeradius-eap.so(+0x6c51)[0x7f9a7bf40c51]
> /usr/local/freeradius/3.0.x-20170328/lib/libfreeradius-eap.so(eaptls_process+0x3aa)[0x7f9a7bf410b2]
> /usr/local/freeradius/3.0.x-20170328/lib/rlm_eap_peap.so(+0x1f9c)[0x7f9a75c80f9c]
> /usr/local/freeradius/3.0.x-20170328/lib/rlm_eap.so(+0x40f5)[0x7f9a76ab70f5]
> /usr/local/freeradius/3.0.x-20170328/lib/rlm_eap.so(eap_method_select+0x431)[0x7f9a76ab7add]
> /usr/local/freeradius/3.0.x-20170328/lib/rlm_eap.so(+0x2d01)[0x7f9a76ab5d01]
> /usr/local/freeradius/current/sbin/radiusd[0x428d3c]
> /usr/local/freeradius/current/sbin/radiusd[0x4293e9]
> /usr/local/freeradius/current/sbin/radiusd[0x428efd]
> /usr/local/freeradius/current/sbin/radiusd[0x429e1d]
> /usr/local/freeradius/current/sbin/radiusd(modcall+0xa2)[0x42ab62]
> /usr/local/freeradius/current/sbin/radiusd(indexed_modcall+0x363)[0x426374]
> /usr/local/freeradius/current/sbin/radiusd(process_authenticate+0x22)[0x428639]
> /usr/local/freeradius/current/sbin/radiusd[0x40f7ed]
> /usr/local/freeradius/current/sbin/radiusd(rad_authenticate+0x51e)[0x410118]
> /usr/local/freeradius/current/sbin/radiusd[0x43d74c]
> /usr/local/freeradius/current/sbin/radiusd[0x439382]
> /lib64/libpthread.so.0(+0x7dc5)[0x7f9a7aa1fdc5]
> /lib64/libc.so.6(clone+0x6d)[0x7f9a7a51773d]
> Calling: gdb -silent -x /usr/local/freeradius/config/raddb/panic.gdb /usr/local/freeradius/current/sbin/radiusd 18297 2>&1 | tee /var/log/radius/gdb-radiusd-18297.log
> Tue Mar 28 09:56:14 2017 : Info: Need 1 more connections to reach 5 spares
> Tue Mar 28 09:56:14 2017 : Info: rlm_sql (sql-null-acct-erx): Opening additional connection (4), 1 of 96 pending slots used
> Panic action exited with 0
> _EXIT(0) CALLED src/lib/debug.c[743]
> Tue Mar 28 10:15:49 2017 : Error: Cannot update core dump limit: Operation not permitted
> Tue Mar 28 10:15:49 2017 : Info: Core dumps are enabled
> Tue Mar 28 10:15:49 2017 : Info: Debugger not attached
>
>
>
>
> -
> List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
>

--
Stefan WINTER
Ingenieur de Recherche
Fondation RESTENA - Réseau Téléinformatique de l'Education Nationale et
de la Recherche
2, avenue de l'Université
L-4365 Esch-sur-Alzette

Tel: +352 424409 1
Fax: +352 422473

PGP key updated to 4096 Bit RSA - I will encrypt all mails if the
recipient's key is known to me

http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xC0DE6A358A39DC66

-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: another crash on 3.0.x

Stefan Winter-4
Hi,

followed by a silent night without issues. Looks all good to me!

Waiting for the 3.0.14 tagging now :-)

Stefan

Am 18.05.2017 um 16:03 schrieb Stefan Winter:

> Hi,
>
> I just tried current 3.0.x and have more than an hour of runtime without
> this issue re-occuring.
>
> So, whatever it was, looks like it's fixed. I'll keep 3.0.x 3047fc6
> running in production under close supervision. If it explodes, I'll sure
> let the list know.
>
> Greetings,
>
> Stefan Winter
>
> Am 28.03.2017 um 10:22 schrieb Stefan Winter:
>> Hi,
>>
>> with that newest 3.0.x from today, I just got a crash (after some 15 min of normal operation, so nothing about stale EAP sessions this time).
>>
>> The directory in question exists (/var/log/radius/tlscache-eduroam-users/) but possibly there's simply no stored session if it's a fresh auth.
>>
>> I (again) don't have an easy way to get a full backtrace as this is happening on a live system, but maybe the below gives a hint already.
>>
>> Greetings,
>>
>> Stefan Winter
>>
>> Tue Mar 28 09:56:08 2017 : Info: rlm_sql (sql-null-acct-erx): Opening additional connection (3), 1 of 97 pending slots used
>> Tue Mar 28 09:56:11 2017 : Info: (12277) eap_peap:   This means you need to read the PREVIOUS messages in the debug output
>> Tue Mar 28 09:56:11 2017 : Info: (12277) eap_peap:   to find out the reason why the user was rejected
>> Tue Mar 28 09:56:11 2017 : Info: (12277) eap_peap:   Look for "reject" or "fail".  Those earlier messages will tell you
>> Tue Mar 28 09:56:11 2017 : Info: (12277) eap_peap:   what went wrong, and how to fix the problem
>> Tue Mar 28 09:56:11 2017 : Error: Couldn't open /var/log/radius/tlscache-eduroam-users/acd33e166c8284cf667a3990565aac2efa85f6ea8c0f813fba87448829b947e5.vps for reading: No such file or directory
>> CAUGHT SIGNAL: Segmentation fault
>> Backtrace of last 29 frames:
>> /usr/local/freeradius/3.0.x-20170328/lib/libfreeradius-radius.so(fr_fault+0x115)[0x7f9a7c155461]
>> /lib64/libpthread.so.0(+0xf370)[0x7f9a7aa27370]
>> /lib64/libssl.so.10(+0x47c92)[0x7f9a7b929c92]
>> /lib64/libssl.so.10(SSL_CTX_add_session+0xe7)[0x7f9a7b92aaf7]
>> /lib64/libssl.so.10(ssl_get_prev_session+0x40c)[0x7f9a7b92afcc]
>> /lib64/libssl.so.10(ssl3_get_client_hello+0x227)[0x7f9a7b8ffbd7]
>> /lib64/libssl.so.10(ssl3_accept+0x95d)[0x7f9a7b90481d]
>> /lib64/libssl.so.10(ssl23_accept+0x1c8)[0x7f9a7b912508]
>> /lib64/libssl.so.10(ssl23_read+0x32)[0x7f9a7b9136a2]
>> /usr/local/freeradius/current/sbin/radiusd(tls_handshake_recv+0x110)[0x45058c]
>> /usr/local/freeradius/3.0.x-20170328/lib/libfreeradius-eap.so(+0x6c51)[0x7f9a7bf40c51]
>> /usr/local/freeradius/3.0.x-20170328/lib/libfreeradius-eap.so(eaptls_process+0x3aa)[0x7f9a7bf410b2]
>> /usr/local/freeradius/3.0.x-20170328/lib/rlm_eap_peap.so(+0x1f9c)[0x7f9a75c80f9c]
>> /usr/local/freeradius/3.0.x-20170328/lib/rlm_eap.so(+0x40f5)[0x7f9a76ab70f5]
>> /usr/local/freeradius/3.0.x-20170328/lib/rlm_eap.so(eap_method_select+0x431)[0x7f9a76ab7add]
>> /usr/local/freeradius/3.0.x-20170328/lib/rlm_eap.so(+0x2d01)[0x7f9a76ab5d01]
>> /usr/local/freeradius/current/sbin/radiusd[0x428d3c]
>> /usr/local/freeradius/current/sbin/radiusd[0x4293e9]
>> /usr/local/freeradius/current/sbin/radiusd[0x428efd]
>> /usr/local/freeradius/current/sbin/radiusd[0x429e1d]
>> /usr/local/freeradius/current/sbin/radiusd(modcall+0xa2)[0x42ab62]
>> /usr/local/freeradius/current/sbin/radiusd(indexed_modcall+0x363)[0x426374]
>> /usr/local/freeradius/current/sbin/radiusd(process_authenticate+0x22)[0x428639]
>> /usr/local/freeradius/current/sbin/radiusd[0x40f7ed]
>> /usr/local/freeradius/current/sbin/radiusd(rad_authenticate+0x51e)[0x410118]
>> /usr/local/freeradius/current/sbin/radiusd[0x43d74c]
>> /usr/local/freeradius/current/sbin/radiusd[0x439382]
>> /lib64/libpthread.so.0(+0x7dc5)[0x7f9a7aa1fdc5]
>> /lib64/libc.so.6(clone+0x6d)[0x7f9a7a51773d]
>> Calling: gdb -silent -x /usr/local/freeradius/config/raddb/panic.gdb /usr/local/freeradius/current/sbin/radiusd 18297 2>&1 | tee /var/log/radius/gdb-radiusd-18297.log
>> Tue Mar 28 09:56:14 2017 : Info: Need 1 more connections to reach 5 spares
>> Tue Mar 28 09:56:14 2017 : Info: rlm_sql (sql-null-acct-erx): Opening additional connection (4), 1 of 96 pending slots used
>> Panic action exited with 0
>> _EXIT(0) CALLED src/lib/debug.c[743]
>> Tue Mar 28 10:15:49 2017 : Error: Cannot update core dump limit: Operation not permitted
>> Tue Mar 28 10:15:49 2017 : Info: Core dumps are enabled
>> Tue Mar 28 10:15:49 2017 : Info: Debugger not attached
>>
>>
>>
>>
>> -
>> List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
>>
>
>
>
>
> -
> List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
>

--
Stefan WINTER
Ingenieur de Recherche
Fondation RESTENA - Réseau Téléinformatique de l'Education Nationale et
de la Recherche
2, avenue de l'Université
L-4365 Esch-sur-Alzette

Tel: +352 424409 1
Fax: +352 422473

PGP key updated to 4096 Bit RSA - I will encrypt all mails if the
recipient's key is known to me

http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xC0DE6A358A39DC66

-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: another crash on 3.0.x

Alan DeKok-2

> On May 19, 2017, at 2:25 AM, Stefan Winter <[hidden email]> wrote:
>
> followed by a silent night without issues. Looks all good to me!

  Great, thanks.

> Waiting for the 3.0.14 tagging now :-)

  We'll do that early next week.

  Alan DeKok.


-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html