Performance tweaking and testing.

[Arne.Olav.Kjosnes]

Hi all,

We have 1500 customers connected to our PPPoE servers, to autenticate we have 2 freeradius servers connected to a mssql server.
The radius servers don't autenticate fast enough, our PPPoE server starts to hang when there are to many radius autentications idle/waiting for a answer from the radius server.

Does anyone got a tip on how to improve performance on the radius servers?

And how to test preformance?
-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html

[Michael Mitchell]

>
> We have 1500 customers connected to our PPPoE servers, to autenticate we
> have 2 freeradius servers connected to a mssql server.

How many authentications per second are you expecting?

With decent hardware you should be able to authenticate all 1500 within
a couple of seconds. I've tested our development Sun V440 to over 600
authentications per second using openLDAP as a backend with 1,000,000+
entries and a random spread of usernames across those 1,000,000 entries.
My client was the limiting factor though, I couldn't max out the CPU of
the RADIUS server.


>
> Does anyone got a tip on how to improve performance on the radius servers?

The biggest bottleneck is likely to be your database. Check your
indexes, etc.

Homename lookups may be an issue too if the server is waiting to for DNS
lookups. Not sure if this is an issue at request processing time or just
at startup. Try turning it off (radiusd.conf) and see if it makes a
difference.

Only log what is necessary. Are you logging request and reply packets?
If so, do you *need* to?

If authentication is *REALLY* slow (ie more than a couple of seconds per
request), run the server in debug mode and you may be able to see which
operations are taking the time.

>
> And how to test preformance?

radclient can be used to send multiple requests to a radius server.

I wrote myself some rough perl scripts to perform authentication and
accounting "load" testing and report on average number of requests
handled per second, etc.

Hope that helps a little

regards,
Mike

-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html

[Arne.Olav.Kjosnes]

Thanks for the reply.

Many good tips, I have added som info/questions/answers under some of them.

Another question:
What did you set your max_requests, start_servers and max_servers to?

[hidden email] wrote on 11.06.2005 14:01:58:
>
> >
> > We have 1500 customers connected to our PPPoE servers, to autenticate we
> > have 2 freeradius servers connected to a mssql server.
>
> How many authentications per second are you expecting?
>
> With decent hardware you should be able to authenticate all 1500 within
> a couple of seconds. I've tested our development Sun V440 to over 600
> authentications per second using openLDAP as a backend with 1,000,000+
> entries and a random spread of usernames across those 1,000,000 entries.
> My client was the limiting factor though, I couldn't max out the CPU of
> the RADIUS server.

at least 700, if a server goes down, the other must take over. Or if we have to do a emergency reboot.
It should handle at least that many.

>
>
> >
> > Does anyone got a tip on how to improve performance on the radius servers?
>
> The biggest bottleneck is likely to be your database. Check your
> indexes, etc.

I will try some more tweaking, but I would like to have a test tool first.
So I could see the differences.

>
> Homename lookups may be an issue too if the server is waiting to for DNS
> lookups. Not sure if this is an issue at request processing time or just
> at startup. Try turning it off (radiusd.conf) and see if it makes a
> difference.

It is turned off

>
> Only log what is necessary. Are you logging request and reply packets?
> If so, do you *need* to?

I have removed everything that I don't feel is necesarry.
>
> If authentication is *REALLY* slow (ie more than a couple of seconds per
> request), run the server in debug mode and you may be able to see which
> operations are taking the time.

It is not that slow, but I will try debug mode anyway to see if I can see any errors.

>
> >
> > And how to test preformance?
>
> radclient can be used to send multiple requests to a radius server.
>
> I wrote myself some rough perl scripts to perform authentication and
> accounting "load" testing and report on average number of requests
> handled per second, etc.

So did I, but it ended up with the "test machine" taking up 100% cpu and not beeing able to authenticate more than 20 pr. sec.
I should work on my coding skills ;)

It would be great if you could share that script with me.
>
> Hope that helps a little
>
> regards,
> Mike
>
> -
> List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html

-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html

[Alan DeKok]

[hidden email] wrote:
> What did you set your max_requests, start_servers and max_servers to?

  With only 1500 users, the defaults should be OK.

> > My client was the limiting factor though, I couldn't max out the CPU of
> > the RADIUS server.
>
> at least 700, if a server goes down, the other must take over. Or if we
> have to do a emergency reboot.
> It should handle at least that many.

  700 requests/s?  I don't think that's necessary.  With only 1500
users, even 300-400/s would be sufficient.

  The server can do that on pretty much any hardware.  If you're
seeing problems, it's most likely because the database is slow.

  Can you describe *exactly* what problemns you're seeing?  Saying the
server is slow is a start, but isn't enough to really know what's
going on.

> I will try some more tweaking, but I would like to have a test tool first.
> So I could see the differences.

  radclient.  It will send as many requests as you want, as fast as
you want.

> So did I, but it ended up with the "test machine" taking up 100% cpu and
> not beeing able to authenticate more than 20 pr. sec.

  I don't see why.  radclient can send as many RADIUS packets as you
want as fast as you want.  External scripts aren't necessary.

  Alan DeKok.

-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html

[Nicolas Baradakis]

[hidden email] wrote:

> I will try some more tweaking, but I would like to have a test tool first.
> So I could see the differences.

Install a CVS snapshot of FreeRADIUS on the machine which runs the
client. New options -p and -n have been added to radclient to
respectively send 'p' packets in parallel or 'n' packets per second.
It's very convenient to run stress tests on the server.

--
Nicolas Baradakis
-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html

[Michael Mitchell]

Nicolas Baradakis wrote:

> [hidden email] wrote:
>
>
>>I will try some more tweaking, but I would like to have a test tool first.
>>So I could see the differences.
>
>
> Install a CVS snapshot of FreeRADIUS on the machine which runs the
> client. New options -p and -n have been added to radclient to
> respectively send 'p' packets in parallel or 'n' packets per second.
> It's very convenient to run stress tests on the server.
>

Only issue with radclient is it uses a single set of attributes for
every request. With a custom script I can run whatever spread of
usernames I want so that I'm not hitting the same (cached in LDAP) user
every single time. Comes down to what you're trying to test really -
best possible performance, or a more "realistic" representative of your
entire system.

-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html

[Alan DeKok]

Michael Mitchell <[hidden email]> wrote:
> Only issue with radclient is it uses a single set of attributes for
> every request.

  Nope.  It reads any number of requests from any number of files,
caches them, and then starts sending data to the server.

  It's not quite the same as dynamic requests, but you should be able
to cache 10^6 requests or so (if you have the memory), which should be
nearly the same thing.

  Alan DeKok.
-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html

[Michael Mitchell]

>
>   Nope.  It reads any number of requests from any number of files,
> caches them, and then starts sending data to the server.

Ahh well thats perfect then!


-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html