core dumps not working on FreeBSD host

classic Classic list List threaded Threaded
5 messages Options
| Threaded
Open this post in threaded view
|

core dumps not working on FreeBSD host

Marek Zarychta
Dear list,

since upgrade from 3.0.19 to 3.0.20 I am occasionally getting errors
like this one:

pid 23749 (radiusd), jid 1, uid 133: exited on signal 11

This issue is somehow related to network / connectivity problems and
looks like quite a new problem (I have not seen this in 3.0.19 and
earlier releases). It can't be easily triggered, so I am searching for
the culprit. To find it I have tried to enable core dumps on FreeBSD for
radiusd but without success. I have set allow_core_dumps = yes in
radiusd.conf file. Also these sysctls are set:
kern.sugid_coredump=1
kern.coredump=1
kern.corefile=/tmp/%N.%P.core

There is no problem with getting core dumps for other daemons running on
this host, also those running with suid/sgid enabled like bind or nsd,
so the issue looks more like daemon related problem than OS problem.

Any help would be greatly appreciated!

--
Marek Zarychta


-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html

signature.asc (499 bytes) Download Attachment
| Threaded
Open this post in threaded view
|

Re: core dumps not working on FreeBSD host

Marek Zarychta
W dniu 05.02.2020 o 14:25, Marek Zarychta pisze:

> Dear list,
>
> since upgrade from 3.0.19 to 3.0.20 I am occasionally getting errors
> like this one:
>
> pid 23749 (radiusd), jid 1, uid 133: exited on signal 11
>
> This issue is somehow related to network / connectivity problems and
> looks like quite a new problem (I have not seen this in 3.0.19 and
> earlier releases). It can't be easily triggered, so I am searching for
> the culprit. To find it I have tried to enable core dumps on FreeBSD for
> radiusd but without success. I have set allow_core_dumps = yes in
> radiusd.conf file. Also these sysctls are set:
> kern.sugid_coredump=1
> kern.coredump=1
> kern.corefile=/tmp/%N.%P.core
>
> There is no problem with getting core dumps for other daemons running on
> this host, also those running with suid/sgid enabled like bind or nsd,
> so the issue looks more like daemon related problem than OS problem.
>
> Any help would be greatly appreciated!
>
>
> -
> List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
>
Running radiusd -X and going through debug output I found these lines:

Changing value of PR_DUMPABLE not supported on this system
Core dumps are enabled

Which probably according to
https://github.com/FreeRADIUS/freeradius-server/blob/v3.0.x/src/lib/debug.c#L480

mean that core dumps are not really supported while running FreeRADIUS
on this OS ?


--
Marek Zarychta


-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html

signature.asc (499 bytes) Download Attachment
| Threaded
Open this post in threaded view
|

Re: core dumps not working on FreeBSD host

Alan DeKok-2
On Feb 5, 2020, at 12:17 PM, Marek Zarychta <[hidden email]> wrote:

> Running radiusd -X and going through debug output I found these lines:
>
> Changing value of PR_DUMPABLE not supported on this system
> Core dumps are enabled
>
> Which probably according to
> https://github.com/FreeRADIUS/freeradius-server/blob/v3.0.x/src/lib/debug.c#L480
>
> mean that core dumps are not really supported while running FreeRADIUS
> on this OS ?

  I suppose not.  We don't currently had a FreeBSD system for testing.  I'm sure it's fixable, but it means spending some time trying to understand what we need to do.

  Alan DeKok.


-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
| Threaded
Open this post in threaded view
|

Re: core dumps not working on FreeBSD host

Jorge Pereira-2
In reply to this post by Marek Zarychta
Hi Marek,

  If you could trigger it easily, maybe you could just start the process with GDB and after catching the crash just uses the command “bt full all threads” to show the entire stack trace and send it to us or create a ticket https://github.com/FreeRADIUS/freeradius-server/issues/new/choose <https://github.com/FreeRADIUS/freeradius-server/issues/new/choose>

It would be good if you could do that.

--
Jorge Pereira
[hidden email]



> On 5 Feb 2020, at 10:25, Marek Zarychta <[hidden email]> wrote:
>
> Dear list,
>
> since upgrade from 3.0.19 to 3.0.20 I am occasionally getting errors
> like this one:
>
> pid 23749 (radiusd), jid 1, uid 133: exited on signal 11
>
> This issue is somehow related to network / connectivity problems and
> looks like quite a new problem (I have not seen this in 3.0.19 and
> earlier releases). It can't be easily triggered, so I am searching for
> the culprit. To find it I have tried to enable core dumps on FreeBSD for
> radiusd but without success. I have set allow_core_dumps = yes in
> radiusd.conf file. Also these sysctls are set:
> kern.sugid_coredump=1
> kern.coredump=1
> kern.corefile=/tmp/%N.%P.core
>
> There is no problem with getting core dumps for other daemons running on
> this host, also those running with suid/sgid enabled like bind or nsd,
> so the issue looks more like daemon related problem than OS problem.
>
> Any help would be greatly appreciated!
>
> --
> Marek Zarychta
>
> -
> List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html

-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
| Threaded
Open this post in threaded view
|

Re: core dumps not working on FreeBSD host

Marek Zarychta
W dniu 05.02.2020 o 19:20, Jorge Pereira pisze:
> Hi Marek,
>
>   If you could trigger it easily, maybe you could just start the process with GDB and after catching the crash just uses the command “bt full all threads” to show the entire stack trace and send it to us or create a ticket https://github.com/FreeRADIUS/freeradius-server/issues/new/choose <https://github.com/FreeRADIUS/freeradius-server/issues/new/choose>
>
> It would be good if you could do that.
>
> --
> Jorge Pereira
> [hidden email]

I have been using freeradius3-3.0.20 since Nov 26th 2019 on both
systems, the issue has occurred only 3 or 4 times since then, so it
can't be triggered easily. I believe that running the process in the
debugger for weeks is not an acceptable scenario. I suspect that some
network related problems triggered the issue but it's not easily
reproducible. I must admit that so far all previous versions survived
network outages easily.

I am running FreeRADIUS on FreeBSD for 10+ years, never encountered
strictly OS related issues since both FreeRADIUS and FreeBSD are great
pieces of software. I am grateful and really appreciate your work guys
as well as the ability to get your support here.

With kind regards,


--
Marek Zarychta


-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html

signature.asc (499 bytes) Download Attachment