3.0.21 consistency check failed

classic Classic list List threaded Threaded
6 messages Options
| Threaded
Open this post in threaded view
|

3.0.21 consistency check failed

adrian.p.smith
We have recently attempted to upgrade from 3.0.15 to 3.0.21 but are seeing very frequent crashes. Config unchanged from 3.0.15.

Any ideas appreciated.

Regards,

Adrian Smith

Examples from the radius.log:

CONSISTENCY CHECK FAILED src/lib/cursor.c[219]: VALUE_PAIR "BTOpenzone-Roaming-Direction" length 42696464 is greater than char buffer length 7

SOFT ASSERT FAILED src/lib/pair.c[2341]: 0
CAUGHT SIGNAL: Aborted
Backtrace of last 15 frames:
/opt/freeradius-server-3.0.21/lib/libfreeradius-radius.so(fr_fault+0x115)[0x7fab75ae7490]
/opt/freeradius-server-3.0.21/lib/libfreeradius-radius.so(fr_assert_cond+0x4c)[0x7fab75ae7f50]
/opt/freeradius-server-3.0.21/lib/libfreeradius-radius.so(fr_pair_verify+0x3af)[0x7fab75af8c52]
/opt/freeradius-server-3.0.21/lib/libfreeradius-radius.so(fr_cursor_next+0x58)[0x7fab75ae641f]
/opt/freeradius-server-3.0.21/lib/libfreeradius-radius.so(fr_pair_list_verify+0x151)[0x7fab75af9169]
/opt/freeradius-server-3.0.21/lib/libfreeradius-server.so(+0x25482)[0x7fab75d4c482]
/opt/freeradius-server-3.0.21/lib/libfreeradius-server.so(verify_request+0x16c)[0x7fab75d4c5f8]
/opt/freeradius-server-3.0.21/sbin/radiusd[0x43d3f0]
/opt/freeradius-server-3.0.21/sbin/radiusd[0x43d1f6]
/opt/freeradius-server-3.0.21/lib/libfreeradius-radius.so(fr_event_run+0x144)[0x7fab75b0f7d4]
/opt/freeradius-server-3.0.21/lib/libfreeradius-radius.so(fr_event_loop+0x509)[0x7fab75b10117]
/opt/freeradius-server-3.0.21/sbin/radiusd(radius_event_process+0x26)[0x4492d5]
/opt/freeradius-server-3.0.21/sbin/radiusd(main+0xcbb)[0x43274b]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x3a0081ed1d]
/opt/freeradius-server-3.0.21/sbin/radiusd[0x40f489]
No panic action set

and

Mon Aug 24 01:20:40 2020 : Error: CONSISTENCY CHECK FAILED src/main/process.c[639]: Expected RADIUS_PACKET request to be parented by 0xd1f5d0 (REQUEST), but parented by 0xd04680 (null_context)
Talloc chunk lineage:
0xd1f230 (RADIUS_PACKET) < 0xd04680 (null_context)
Talloc context level 0:
full talloc report on 'null_context' (total 466556 bytes in 3189 blocks)
    session-state                  contains     14 bytes in   2 blocks (ref 0) 0xd8c690
        .name                          contains     14 bytes in   1 blocks (ref 0) 0xd79b20
    acct_listener_pool             contains  15485 bytes in 101 blocks (ref 0) 0xd77ab0
        REQUEST                        contains   7277 bytes in 100 blocks (ref 0) 0xd77e30
            RADIUS_PACKET                  contains   3340 bytes in  48 blocks (ref 0) 0x7f090001f090
                uint8_t                        contains    481 bytes in   1 blocks (ref 0) 0x7f090000fd40
<snip>
Mon Aug 24 01:20:40 2020 : Error: ASSERT FAILED src/main/util.c[1095]: 0
CAUGHT SIGNAL: Aborted
Backtrace of last 12 frames:
/opt/freeradius-server-3.0.21/lib/libfreeradius-radius.so(fr_fault+0x115)[0x7f092d5aa490]
/opt/freeradius-server-3.0.21/lib/libfreeradius-server.so(rad_assert_fail+0x46)[0x7f092d80e438]
/opt/freeradius-server-3.0.21/lib/libfreeradius-server.so(+0x25437)[0x7f092d80f437]
/opt/freeradius-server-3.0.21/lib/libfreeradius-server.so(verify_request+0x10a)[0x7f092d80f596]
/opt/freeradius-server-3.0.21/sbin/radiusd[0x43d3f0]
/opt/freeradius-server-3.0.21/sbin/radiusd[0x43d1f6]
/opt/freeradius-server-3.0.21/lib/libfreeradius-radius.so(fr_event_run+0x144)[0x7f092d5d27d4]
/opt/freeradius-server-3.0.21/lib/libfreeradius-radius.so(fr_event_loop+0x509)[0x7f092d5d3117]
/opt/freeradius-server-3.0.21/sbin/radiusd(radius_event_process+0x26)[0x4492d5]
/opt/freeradius-server-3.0.21/sbin/radiusd(main+0xcbb)[0x43274b]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x3a0081ed1d]
/opt/freeradius-server-3.0.21/sbin/radiusd[0x40f489]
No panic action set

and

CONSISTENCY CHECK FAILED src/lib/cursor.c[219]: VALUE_PAIR "NAS-Identifier" char buffer is not parented by VALUE_PAIR 0x7fc3f0014480, instead parented by 0x231dee0 (null_context)

SOFT ASSERT FAILED src/lib/pair.c[2358]: 0
CAUGHT SIGNAL: Aborted
Backtrace of last 15 frames:
/opt/freeradius-server-3.0.21/lib/libfreeradius-radius.so(fr_fault+0x115)[0x7fc46fbd8490]
/opt/freeradius-server-3.0.21/lib/libfreeradius-radius.so(fr_assert_cond+0x4c)[0x7fc46fbd8f50]
/opt/freeradius-server-3.0.21/lib/libfreeradius-radius.so(fr_pair_verify+0x4cc)[0x7fc46fbe9d6f]
/opt/freeradius-server-3.0.21/lib/libfreeradius-radius.so(fr_cursor_next+0x58)[0x7fc46fbd741f]
/opt/freeradius-server-3.0.21/lib/libfreeradius-radius.so(fr_pair_list_verify+0x151)[0x7fc46fbea169]
/opt/freeradius-server-3.0.21/lib/libfreeradius-server.so(+0x25482)[0x7fc46fe3d482]
/opt/freeradius-server-3.0.21/lib/libfreeradius-server.so(verify_request+0x16c)[0x7fc46fe3d5f8]
/opt/freeradius-server-3.0.21/sbin/radiusd[0x43d3f0]
/opt/freeradius-server-3.0.21/sbin/radiusd[0x43d1f6]
/opt/freeradius-server-3.0.21/lib/libfreeradius-radius.so(fr_event_run+0x144)[0x7fc46fc007d4]
/opt/freeradius-server-3.0.21/lib/libfreeradius-radius.so(fr_event_loop+0x509)[0x7fc46fc01117]
/opt/freeradius-server-3.0.21/sbin/radiusd(radius_event_process+0x26)[0x4492d5]
/opt/freeradius-server-3.0.21/sbin/radiusd(main+0xcbb)[0x43274b]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x33d121ed1d]
/opt/freeradius-server-3.0.21/sbin/radiusd[0x40f489]
No panic action set

and

Mon Aug 24 01:20:49 2020 : Error: CONSISTENCY CHECK FAILED src/main/process.c[2919]: Expected RADIUS_PACKET request to be parented by 0x1878490 (REQUEST), but parented by 0x184bee0 (null_context)
Talloc chunk lineage:
0x1878170 (RADIUS_PACKET) < 0x184bee0 (null_context)
Talloc context level 0:
full talloc report on 'null_context' (total  64956 bytes in 465 blocks)
    session-state                  contains     14 bytes in   2 blocks (ref 0) 0x1873070
        .name                          contains     14 bytes in   1 blocks (ref 0) 0x187e190
    acct_listener_pool             contains  15323 bytes in  94 blocks (ref 0) 0x196f200
<snip>
Mon Aug 24 01:20:49 2020 : Error: ASSERT FAILED src/main/util.c[1095]: 0
CAUGHT SIGNAL: Aborted
Backtrace of last 9 frames:
/opt/freeradius-server-3.0.21/lib/libfreeradius-radius.so(fr_fault+0x115)[0x7f74c5e88490]
/opt/freeradius-server-3.0.21/lib/libfreeradius-server.so(rad_assert_fail+0x46)[0x7f74c60ec438]
/opt/freeradius-server-3.0.21/lib/libfreeradius-server.so(+0x25437)[0x7f74c60ed437]
/opt/freeradius-server-3.0.21/lib/libfreeradius-server.so(verify_request+0x10a)[0x7f74c60ed596]
/opt/freeradius-server-3.0.21/sbin/radiusd[0x44221c]
/opt/freeradius-server-3.0.21/sbin/radiusd[0x43f5a3]
/opt/freeradius-server-3.0.21/sbin/radiusd[0x43af5c]
/lib64/libpthread.so.0(+0x33d1607aa1)[0x7f74c5840aa1]
/lib64/libc.so.6(clone+0x6d)[0x33d12e8bcd]
No panic action set

And

Mon Aug 24 01:22:08 2020 : Error: CONSISTENCY CHECK FAILED src/main/process.c[1159]: Expected RADIUS_PACKET request to be parented by 0xe77dc0 (REQUEST), but parented by 0xd94ee0 (null_context)
Talloc chunk lineage:
0xe77ab0 (RADIUS_PACKET) < 0xd94ee0 (null_context)
Talloc context level 0:
full talloc report on 'null_context' (total 594675 bytes in 3880 blocks)
    session-state                  contains     14 bytes in   2 blocks (ref 0) 0xe21380
        .name                          contains     14 bytes in   1 blocks (ref 0) 0xda9720
    acct_listener_pool             contains  16303 bytes in 114 blocks (ref 0) 0xda3b10
        REQUEST                        contains   8095 bytes in 113 blocks (ref 0) 0xda3ed0
            RADIUS_PACKET                  contains    258 bytes in   4 blocks (ref 0) 0xe18100
                VALUE_PAIR                     contains     82 bytes in   2 blocks (ref 0) 0x7f55f00076f0
                    uint8_t                        contains      2 bytes in   1 blocks (ref 0) 0x7f55f0006170
                src/lib/radius.c:495           contains     24 bytes in   1 blocks (ref 0) 0xda9370
            RADIUS_PACKET                  contains   3610 bytes in  52 blocks (ref 0) 0x7f5610028b50n

and

Mon Aug 24 01:22:11 2020 : Error: ASSERT FAILED src/main/util.c[566]: !request->in_request_hash
CAUGHT SIGNAL: Aborted
Backtrace of last 17 frames:
/opt/freeradius-server-3.0.21/lib/libfreeradius-radius.so(fr_fault+0x115)[0x7f31e3310490]
/opt/freeradius-server-3.0.21/lib/libfreeradius-server.so(rad_assert_fail+0x46)[0x7f31e3574438]
/opt/freeradius-server-3.0.21/lib/libfreeradius-server.so(+0x24481)[0x7f31e3574481]
/usr/lib64/libtalloc.so.2(+0x2dbe)[0x7f31e2ee2dbe]
/usr/lib64/libtalloc.so.2(+0x2c4b)[0x7f31e2ee2c4b]
/usr/lib64/libtalloc.so.2(+0x2c4b)[0x7f31e2ee2c4b]
/usr/lib64/libtalloc.so.2(+0x2c4b)[0x7f31e2ee2c4b]
/usr/lib64/libtalloc.so.2(_talloc_free+0xfb)[0x7f31e2ee32db]
/opt/freeradius-server-3.0.21/sbin/radiusd[0x43d2e2]
/opt/freeradius-server-3.0.21/sbin/radiusd[0x43db8c]
/opt/freeradius-server-3.0.21/sbin/radiusd[0x43d1f6]
/opt/freeradius-server-3.0.21/lib/libfreeradius-radius.so(fr_event_run+0x144)[0x7f31e33387d4]
/opt/freeradius-server-3.0.21/lib/libfreeradius-radius.so(fr_event_loop+0x509)[0x7f31e3339117]
/opt/freeradius-server-3.0.21/sbin/radiusd(radius_event_process+0x26)[0x4492d5]
/opt/freeradius-server-3.0.21/sbin/radiusd(main+0xcbb)[0x43274b]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x33d121ed1d]
/opt/freeradius-server-3.0.21/sbin/radiusd[0x40f489]
No panic action set
-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
| Threaded
Open this post in threaded view
|

Re: 3.0.21 consistency check failed

Alan DeKok-2
On Aug 24, 2020, at 4:31 AM, [hidden email] wrote:
>
> We have recently attempted to upgrade from 3.0.15 to 3.0.21 but are seeing very frequent crashes. Config unchanged from 3.0.15.

  Are you running custom modules?

> CONSISTENCY CHECK FAILED src/lib/cursor.c[219]: VALUE_PAIR "BTOpenzone-Roaming-Direction" length 42696464 is greater than char buffer length 7

  That's bad.

  As background, we have some additional sanity checks we run on developer builds.  These checks verify that the internal data structures are OK.  The checks slow down the server substantially, but they are enormously useful for debugging.

  If the checks fail, then something is going seriously wrong.  Even worse, if you have many such errors spread all over the server, then *everything* is wrong.

  My guess is that you built 3.0.21 on a system which already had FreeRADIUS installed.  Maybe the headers got confused, as some data structures have changed.

  Build 3.0.21 on a clean system, or use the packages at http://packages.networkradius.com

  That release works in production without any of those errors.

  Alan DeKok.


-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
| Threaded
Open this post in threaded view
|

Re: 3.0.21 consistency check failed

thepete10
 fuck off dickweed

    On Monday, August 24, 2020, 7:28:09 AM EDT, Alan DeKok <[hidden email]> wrote:  
 
 On Aug 24, 2020, at 4:31 AM, [hidden email] wrote:
>
> We have recently attempted to upgrade from 3.0.15 to 3.0.21 but are seeing very frequent crashes. Config unchanged from 3.0.15.

  Are you running custom modules?

> CONSISTENCY CHECK FAILED src/lib/cursor.c[219]: VALUE_PAIR "BTOpenzone-Roaming-Direction" length 42696464 is greater than char buffer length 7

  That's bad.

  As background, we have some additional sanity checks we run on developer builds.  These checks verify that the internal data structures are OK.  The checks slow down the server substantially, but they are enormously useful for debugging.

  If the checks fail, then something is going seriously wrong.  Even worse, if you have many such errors spread all over the server, then *everything* is wrong.

  My guess is that you built 3.0.21 on a system which already had FreeRADIUS installed.  Maybe the headers got confused, as some data structures have changed.

  Build 3.0.21 on a clean system, or use the packages at http://packages.networkradius.com

  That release works in production without any of those errors.

  Alan DeKok.


-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html 
-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
| Threaded
Open this post in threaded view
|

Re: 3.0.21 consistency check failed

Alan DeKok-2
On Aug 24, 2020, at 9:26 AM, thepete10 <[hidden email]> wrote:
...

  That's good for a ban.

  Alan DeKok.

-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
| Threaded
Open this post in threaded view
|

RE: 3.0.21 consistency check failed

adrian.p.smith
In reply to this post by Alan DeKok-2


-----Original Message-----
From: Freeradius-Users <freeradius-users-bounces+adrian.p.smith=[hidden email]> On Behalf Of Alan DeKok
Sent: 24 August 2020 12:27
To: FreeRadius users mailing list <[hidden email]>
Subject: Re: 3.0.21 consistency check failed

On Aug 24, 2020, at 4:31 AM, [hidden email] wrote:
>
> We have recently attempted to upgrade from 3.0.15 to 3.0.21 but are seeing very frequent crashes. Config unchanged from 3.0.15.

  Are you running custom modules?

> CONSISTENCY CHECK FAILED src/lib/cursor.c[219]: VALUE_PAIR "BTOpenzone-Roaming-Direction" length 42696464 is greater than char buffer length 7

  That's bad.

  As background, we have some additional sanity checks we run on developer builds.  These checks verify that the internal data structures are OK.  The checks slow down the server substantially, but they are enormously useful for debugging.

  If the checks fail, then something is going seriously wrong.  Even worse, if you have many such errors spread all over the server, then *everything* is wrong.

  My guess is that you built 3.0.21 on a system which already had FreeRADIUS installed.  Maybe the headers got confused, as some data structures have changed.

  Build 3.0.21 on a clean system, or use the packages at http://packages.networkradius.com

  That release works in production without any of those errors.

  Alan DeKok.



Thanks Alan, will check our build process and give that a go.

Adrian


-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
| Threaded
Open this post in threaded view
|

RE: 3.0.21 consistency check failed

adrian.p.smith
FYI - A fresh build seems to have fixed this issue.



-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html