How make freeradius - rlm_exec run multithreaded?

classic Classic list List threaded Threaded
8 messages Options
| Threaded
Open this post in threaded view
|

How make freeradius - rlm_exec run multithreaded?

Ryan Allen
In FreeRadius, we are trying to use rlm_exec to call an external
script in a multithreaded fashion, but once "exec" is called, it
blocks other requests and only processes one request at a time.  We
need to support multiple users signing in at the same time.  How can
we make this multithreaded/multiprocessed?

Here is a simple example we used to verify the blocking issue:
# In file: raddb/sites-available/default
authorize {
    ...
    exec
    update reply {
        Reply-Message = config:Reply-Message
        State = config:State
    }
    handled
    ...
}

# In file: raddb/mods-enabled/exec
exec {
   program = "/bin/bash -f
/etc/raddb/mods-config/.../execBashCustomAuthorization.sh"
   wait = yes
   input_pairs = request
   output_pairs = config
   shell_escape = yes
   timeout = 30
}

# In file: raddb/mods-config/.../execBashCustomAuthorization.sh
#!/bin/bash
sleep 5
echo "Reply-Message = \"A short message\","
echo "State = \"2496578198c5447a8167fe3fe0398133\","
echo "Response-Packet-Type = \"Access-Challenge\""
exit 0

If we send 5 requests to our freeradius server, we expect all 5
responses to come back at about the same time, but they each respond 5
seconds apart, and it takes about 25 seconds for all to complete.

We know that "wait=yes" makes freeradius wait for the script to finish
and processes what the external script sent to stdout, but we thought
it could still handle multiple requests in parallel.  We can't use
"wait=no" because we need output from the external script.

Here are the details of our runtime environment:
FreeRADIUS Version: 3.0.19
Python Version: 2.7.15

Please let us know if you need any additional environmental or
configuration information.

We thank you in advance for any help you can provide.
-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
| Threaded
Open this post in threaded view
|

Re: [EXT] How make freeradius - rlm_exec run multithreaded?

Brian Julin

Ryan Allen <[hidden email]> wrote:
> In FreeRadius, we are trying to use rlm_exec to call an external
> script in a multithreaded fashion, but once "exec" is called, it
> blocks other requests and only processes one request at a time.  We
> need to support multiple users signing in at the same time.  How can
> we make this multithreaded/multiprocessed?

I was under the impression it already did by virtue of each request being
handled by a worker thread.

Things to check:

A) have you reduced your thread pool for some reason or are you
running in a single threaded debug mode?

B) Are you sending a bunch of the same requests with the
same sequence numbers... FreeRADIUS might think your requests
are retransmissions rather than new requests.  Check the logs.

-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
| Threaded
Open this post in threaded view
|

Re: How make freeradius - rlm_exec run multithreaded?

Alan DeKok-2
In reply to this post by Ryan Allen
On Jan 10, 2020, at 2:46 PM, Ryan Allen <[hidden email]> wrote:
>
> In FreeRadius, we are trying to use rlm_exec to call an external
> script in a multithreaded fashion, but once "exec" is called, it
> blocks other requests and only processes one request at a time.

  The server is multi-threaded.  Each thread runs independently of each other thread.  However, one thread *can* block.  And if it blocks, that thread won't process anything else.

>  We
> need to support multiple users signing in at the same time.  How can
> we make this multithreaded/multiprocessed?
>
> Here is a simple example we used to verify the blocking issue:

  OK...

> If we send 5 requests to our freeradius server, we expect all 5
> responses to come back at about the same time, but they each respond 5
> seconds apart, and it takes about 25 seconds for all to complete.

  Is this in daemon mode, or in debug mode?  Because in debug mode, the server is single threaded.

  You can start multi-threaded debug mode via the following arguments:

radiusd -xx -f -l stdout

  You can then send multiple packets, and verify that it is running multi-threaded.

   If you do that and post that debug output, it *will* show that the server is multi-threaded.

> We know that "wait=yes" makes freeradius wait for the script to finish
> and processes what the external script sent to stdout, but we thought
> it could still handle multiple requests in parallel.

  If you have 5 threads configured, the server will be able to run 5 of those requests at the same time.

  We generally don't recommend using "exec".  It's simple, but forking programs for every packet is *slow*.  Very slow.  If you can use the built-in functionality of the server, it's much better.

  Alan DeKok.


-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
| Threaded
Open this post in threaded view
|

Re: [EXT] How make freeradius - rlm_exec run multithreaded?

Ryan Allen
In reply to this post by Brian Julin
> A1) have you reduced your thread pool for some reason or

# In file: raddb/radiusd.conf, I have:
thread pool {
    start_servers = 5
    max_servers = 32
    min_spare_servers = 3
    max_spare_servers = 10
    max_requests_per_server = 0
    auto_limit_acct = no
}

Is there anything else we need to set in this file or another file?


> A2) are you running in a single threaded debug mode?

When I start freeradius, it is in a docker container, that runs:
# Inside Docker file: /docker-entrypoint.sh
----------------------------
#!/bin/sh
set -e

# this if will check if the first argument is a flag
# but only works if all arguments require a hyphenated flag
# -v; -SL; -f arg; etc will work, but not arg1 arg2
if [ "$#" -eq 0 ] || [ "${1#-}" != "$1" ]; then
    set -- freeradius "$@"
fi

# check for the expected command
if [ "$1" = 'freeradius' ]; then
    shift
    exec freeradius -f "$@"
fi

# many people are likely to call "radiusd" as well, so allow that
if [ "$1" = 'radiusd' ]; then
    shift
    exec freeradius -f "$@"
fi

# else default to run whatever the user wanted like "bash" or "sh"
exec "$@"
----------------------------

In the above file, it runs:
exec freeradius -f "$@"

I don't think it is in debug mode.  When it starts there is only one
line in docker logs which is:
Sat Jan 11 00:00:20 2020 : Info: Ready to process requests

I was running debug mode earlier by changing the line to:
exec freeradius -f -X "$@"
but I removed the -X and still doing one request at a time

Is there anything else we need to do to make sure debug mode is off?


> B) Are you sending a bunch of the same requests with the
> same sequence numbers... FreeRADIUS might think your requests
> are retransmissions rather than new requests.  Check the logs.

I deployed the code to an enterprise Linux server that is hit by an
enterprise server.  Then two of us tried to sign-in at the same time.
We could see that only one request was processed at a time while the
other waited.
The request was sent to an F5 load balancer that would "snat" the IP,
so based on my understanding, both our requests would come from the
same IP, but we are using different usernames, so I would hope it
treats them differently.


Any other ideas?
-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
| Threaded
Open this post in threaded view
|

Re: [EXT] How make freeradius - rlm_exec run multithreaded?

Alan DeKok-2
On Jan 10, 2020, at 7:18 PM, Ryan Allen <[hidden email]> wrote:
>
> Any other ideas?

  Run the server in THREADED debugging mode as I suggested in my previous email.  Nothing else will let you know what *FreeRADIUS* is doing.

  DON'T run it via layers of scripts and wrappers.  That just adds to the confusion.  You're effectively trying to debug layers of other garbage, while blaming it on FreeRADIUS.

  DO run it from the command line in a terminal window.  If necessary, just build and install it WITHOUT docker.  It takes ~5min to install it in a VM from the packages on networkradius.com.

  Alan DeKok.


-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
| Threaded
Open this post in threaded view
|

Re: [EXT] How make freeradius - rlm_exec run multithreaded?

Matthew Newton-3
In reply to this post by Ryan Allen
On Fri, 2020-01-10 at 17:18 -0700, Ryan Allen wrote:
> In the above file, it runs:
> exec freeradius -f "$@"
>
> I don't think it is in debug mode.  When it starts there is only one
> line in docker logs which is:
> Sat Jan 11 00:00:20 2020 : Info: Ready to process requests

You shouldn't see any debug output, normally.


> I deployed the code to an enterprise Linux server that is hit by an
> enterprise server.  Then two of us tried to sign-in at the same time.
> We could see that only one request was processed at a time while the
> other waited.

It works fine with the official freeradius docker image:

https://hub.docker.com/r/freeradius/freeradius-server


$ find .
./raddb
./raddb/clients.conf
./raddb/mods-config
./raddb/mods-config/files
./raddb/mods-config/files/authorize
./raddb/mods-available
./raddb/mods-available/exec
./Dockerfile
$

There's only one file added to the examples given with the docker
instructions:

$ cat ./raddb/mods-available/exec
exec {
        wait = yes
        input_pairs = request
        shell_escape = yes
        timeout = 10
        program = "/bin/sleep 5"
}

Testing:

$ docker build -t rad .

Multi-threaded mode:

$ docker run -d --name radius -p 1812-1813:1812-1813/udp -it rad
0371c99e0257fc2fdf600c9bfd4dcf25eade9b42bc3ec43e94bb4c9e05aa7179
$ time ( radtest bob test 127.0.0.1 0 testing123 & radtest bob test
127.0.0.1 0 testing123 & wait )
...
real 0m5.146s
$ docker container logs radius
...nothing...
$ docker container kill radius
radius
$ docker container rm radius
radius


Whereas in single-threaded debug mode:

$ docker run -d --name radius -p 1812-1813:1812-1813/udp -it rad -X
96144c2969062a838c9bb7b4ceec540533207d35f208550cbce1110648703a2c
$ time ( radtest bob test 127.0.0.1 0 testing123 & radtest bob test
127.0.0.1 0 testing123 & wait )
...
real 0m10.034s
$ docker container logs radius
...lots of logs...
$ docker container kill
radius
radius
$ docker container rm radius
radius



You need to check your Dockerfile, or how you're invoking it.

Take a look at the process list on the docker host; that should give a
quick answer as to whether it's being run with -X or similar. Standard
process in the official docker image for example will be running as
"freeradius -f", e.g.

$ docker run -d --name radius -p 1812-1813:1812-1813/udp -it rad
67f5f6d3a83d988b7cbed7500fc9d106ebbd1e30587d2039f260d08dcbb02d4c
$ ps -ef | grep radius | grep -v grep
systemd+ 10457 10425  0 10:33 pts/0    00:00:00 freeradius -f
$ docker kill radius
radius

Use the parameters Alan gave, and as he said, using exec is really not
recommended anyway.

--
Matthew


-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
| Threaded
Open this post in threaded view
|

Re: [EXT] How make freeradius - rlm_exec run multithreaded?

Ryan Allen
Your ideas helped us find the issue.

We had an outer script that was passing in "-t" as one of the
arguments when we started up freeradius.  The "-t" made freeradius run
in "single threaded" mode.  When we removed the "-t", it worked as
expected.

Thanks for the help!
-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
| Threaded
Open this post in threaded view
|

Re: How make freeradius - rlm_exec run multithreaded?

Alan Buxey
In reply to this post by Ryan Allen
# In file: raddb/mods-config/.../execBashCustomAuthorization.sh
#!/bin/bash
sleep 5
echo "Reply-Message = \"A short message\","
echo "State = \"2496578198c5447a8167fe3fe0398133\","
echo "Response-Packet-Type = \"Access-Challenge\""
exit 0


I guess this is just a demo/example of the sort of thing that the
script will do as otherwise all of what its doing (in terms of setting
values) can be just done
directly in the server.

alan
-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html