FR 3.0.21 : Failed adding proxy socket: Too many open sockets

classic Classic list List threaded Threaded
9 messages Options
| Threaded
Open this post in threaded view
|

FR 3.0.21 : Failed adding proxy socket: Too many open sockets

Arnaud LAURIOU
Hi,

We are upgrading our eduroam proxy server to FR 3.0.21. We have ~550
clients and home_server.

When we start FR, it seems that the number of udp socket keep increases,
we have a lot of log
lines like :
Info:  ... adding new socket proxy address * port 50740

Until this one :
Error: Failed adding proxy socket: Too many open sockets

Then, some proxied requests are still working fine but other ones (new
ones which need a
new socket ?) are rejected :
(365476) Tue Aug 18 15:05:46 2020: ERROR: proxy: Failed allocating Id
for proxied request

With netstat or lsof, udp sockets are opened but not so many :
$ lsof -p <freeradius_pid> | grep UDP | wc -l
259

Any idea how to solve this issue ?

Regards,

Arnaud Lauriou

-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
| Threaded
Open this post in threaded view
|

Re: FR 3.0.21 : Failed adding proxy socket: Too many open sockets

Alan DeKok-2
On Aug 18, 2020, at 9:19 AM, Arnaud LAURIOU <[hidden email]> wrote:
> We are upgrading our eduroam proxy server to FR 3.0.21. We have ~550 clients and home_server.
>
> When we start FR, it seems that the number of udp socket keep increases, we have a lot of log
> lines like :
> Info:  ... adding new socket proxy address * port 50740
>
> Until this one :
> Error: Failed adding proxy socket: Too many open sockets

  Yes, it only supports 256 proxy sockets.

  Are you setting a unique src_ipaddr for each home server?  Because that will definitely increase the number of sockets used.

  There's no issue with 1000 home servers using the same socket.  RADIUS allows it, and FreeRADIUS has been tested with it.

> Then, some proxied requests are still working fine but other ones (new ones which need a
> new socket ?) are rejected :
> (365476) Tue Aug 18 15:05:46 2020: ERROR: proxy: Failed allocating Id for proxied request
>
> With netstat or lsof, udp sockets are opened but not so many :
> $ lsof -p <freeradius_pid> | grep UDP | wc -l
> 259

  That's still a lot of sockets.

> Any idea how to solve this issue ?

  a) allow home servers to share src_ipaddr, which means that they can share sockets

  b) edit src/lib/packet.c, and change MAX_SOCKETS from 256 to 1024, or maybe 2048.  It just has to be a power of 2.

  Alan DeKok.


-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
| Threaded
Open this post in threaded view
|

Re: FR 3.0.21 : Failed adding proxy socket: Too many open sockets

Arnaud LAURIOU
Thank's for your reply.

On 8/18/20 3:25 PM, Alan DeKok wrote:
>> Any idea how to solve this issue ?
>    a) allow home servers to share src_ipaddr, which means that they can share sockets
Just a precision, I added the same src_ipaddr for each home_server but
several udp sockets are
still created :
Tue Aug 18 16:53:02 2020 : Info:  ... adding new socket proxy address
194.57.7.78 port 39252
Tue Aug 18 16:53:51 2020 : Info:  ... adding new socket proxy address
194.57.7.78 port 43055
Tue Aug 18 16:54:48 2020 : Info:  ... adding new socket proxy address
194.57.7.78 port 59876
Tue Aug 18 17:01:21 2020 : Info:  ... adding new socket proxy address
194.57.7.78 port 38393

Is there something else to do to share this src_ipaddr ?

Regards,

Arnaud Lauriou

-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
| Threaded
Open this post in threaded view
|

Re: FR 3.0.21 : Failed adding proxy socket: Too many open sockets

Alan DeKok-2

On Aug 18, 2020, at 11:03 AM, Arnaud LAURIOU <[hidden email]> wrote:
> Just a precision, I added the same src_ipaddr for each home_server but several udp sockets are
> still created :
> Tue Aug 18 16:53:02 2020 : Info:  ... adding new socket proxy address 194.57.7.78 port 39252
> Tue Aug 18 16:53:51 2020 : Info:  ... adding new socket proxy address 194.57.7.78 port 43055
> Tue Aug 18 16:54:48 2020 : Info:  ... adding new socket proxy address 194.57.7.78 port 59876
> Tue Aug 18 17:01:21 2020 : Info:  ... adding new socket proxy address 194.57.7.78 port 38393
>
> Is there something else to do to share this src_ipaddr ?

  No.  If you have not set src_ipaddr before, then it adding it now won't change things.

  I suspect that one of your home servers is down.  Limitations in RADIUS mean that the server can only send 256 packets on one socket before it needs to open a new socket.  So if a home server is down or slow, it might very well use 256 packets & 256 sockets, just to talk to one home server.

  You may have to edit src/lib/packet.c as I suggested before.  But even that may not help if one of the home servers is down.

  Alan DeKok.


-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
| Threaded
Open this post in threaded view
|

Re: FR 3.0.21 : Failed adding proxy socket: Too many open sockets

Arnaud LAURIOU


On 8/18/20 5:07 PM, Alan DeKok wrote:
>
>    I suspect that one of your home servers is down.  Limitations in RADIUS mean that the server can only send 256 packets on one socket before it needs to open a new socket.  So if a home server is down or slow, it might very well use 256 packets & 256 sockets, just to talk to one home server.
>
>    You may have to edit src/lib/packet.c as I suggested before.  But even that may not help if one of the home servers is down.
>

Well, I may have found home server guilty, a broken one. I'm
investigating ...
But :
- I don't understand why we didn't have this issue with our current FR
2.1 proxy using the same home servers list
and having 500 open udp sockets.
- It's difficult to prevent from all home servers down/slow or broken.

Is there a way to mitigate this issue ?
Close udp sockets ?
It seems that once this limit is reached the current sockets are never
closed and all new clients requests are rejected.

Regards,

Arnaud Lauriou

-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
| Threaded
Open this post in threaded view
|

Re: FR 3.0.21 : Failed adding proxy socket: Too many open sockets

Alan DeKok-2
On Aug 19, 2020, at 5:50 AM, Arnaud LAURIOU <[hidden email]> wrote:
> Well, I may have found home server guilty, a broken one. I'm investigating ...
> But :
> - I don't understand why we didn't have this issue with our current FR 2.1 proxy using the same home servers list
> and having 500 open udp sockets.

  v2 is different from v3.

> - It's difficult to prevent from all home servers down/slow or broken.
>
> Is there a way to mitigate this issue ?

  Read proxy.conf.  Look for "max_outstanding".  This is documented.

> Close udp sockets ?
> It seems that once this limit is reached the current sockets are never closed and all new clients requests are rejected.

  Likely because it's still trying to proxy to the home server, as it still thinks that the home server is alive.

  If you fill up the proxy with bad packets, then bad things happen.  Update the configuration to limit the number of packets proxied to that home server, and bad things won't happen.

  Alan DeKok.


-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
| Threaded
Open this post in threaded view
|

Re: FR 3.0.21 : Failed adding proxy socket: Too many open sockets

Arnaud LAURIOU


On 8/19/20 1:50 PM, Alan DeKok wrote:
>
>> - It's difficult to prevent from all home servers down/slow or broken.
>>
>> Is there a way to mitigate this issue ?
>    Read proxy.conf.  Look for "max_outstanding".  This is documented.
Ok, thanks Alan for pointing out that counter.

I have a few questions for which I haven't found answer yet :
Is there good tuning practices ?
Is this counter reset ? For example :
- If a home_server is temporarly dead, is his value reset when it comes
alive again ?
- If a home_server reaches this counter limit, how long will it be unused ?

Regards,

Arnaud Lauriou
-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
| Threaded
Open this post in threaded view
|

Re: FR 3.0.21 : Failed adding proxy socket: Too many open sockets

Alan DeKok-2
On Aug 20, 2020, at 10:46 AM, Arnaud LAURIOU <[hidden email]> wrote:
> I have a few questions for which I haven't found answer yet :
> Is there good tuning practices ?

  Set it to a value which is reasonable.  What "reasonable" means depends on your situation.

  i.e. a busy server may *normally* have thousands of packets outstanding.  A small server may normally have 10 packets outstanding.

> Is this counter reset ? For example :
> - If a home_server is temporarly dead, is his value reset when it comes alive again ?

  If the home server is dead, then it has no outstanding packets.  And therefore the outstanding packet counter is zero.

> - If a home_server reaches this counter limit, how long will it be unused ?

  Until it either responds to packets (which lowers the outstanding counter), or it is marked as dead (which sets the counter to zero).

  i.e. it does what makes sense.

  Alan DeKok.


-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
| Threaded
Open this post in threaded view
|

Re: [EXTERNAL] Re: FR 3.0.21 : Failed adding proxy socket: Too many open sockets

Users mailing list
Add to this that you should care about latency. No point queuing 1000's of extra requests if it means that the queue is too long to process...

That is the time to process everything in the queue > timeout of clients / devices. (Only option if that’s the case is more capacity or optimisation wherever the bottleneck is).

To get AAA tuning right the only option is test it with realistic requests and request rates and see what you get. Get into the habit of doing this for every change and you won't get surprised by sudden performance deltas associated with what on the face of it are trivial configuration updates. Also beware the hockey stick performance case found by graphing request rate vs latency. See a huge latency delta for a tiny delta in request rate and you need to be careful if you are running anywhere near the limit.

Alister.


On 20/08/2020, 15:55, "Freeradius-Users on behalf of Alan DeKok" <freeradius-users-bounces+alister.winfield=[hidden email] on behalf of [hidden email]> wrote:

    On Aug 20, 2020, at 10:46 AM, Arnaud LAURIOU <[hidden email]> wrote:
    > I have a few questions for which I haven't found answer yet :
    > Is there good tuning practices ?

      Set it to a value which is reasonable.  What "reasonable" means depends on your situation.

      i.e. a busy server may *normally* have thousands of packets outstanding.  A small server may normally have 10 packets outstanding.

    > Is this counter reset ? For example :
    > - If a home_server is temporarly dead, is his value reset when it comes alive again ?

      If the home server is dead, then it has no outstanding packets.  And therefore the outstanding packet counter is zero.

    > - If a home_server reaches this counter limit, how long will it be unused ?

      Until it either responds to packets (which lowers the outstanding counter), or it is marked as dead (which sets the counter to zero).

      i.e. it does what makes sense.

      Alan DeKok.


    -
    List info/subscribe/unsubscribe? See https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.freeradius.org%2Flist%2Fusers.html&amp;data=02%7C01%7Calister.winfield%40sky.uk%7C6e6a50a7ffa54150d27a08d845190583%7C68b865d5cf184b2b82a4a4eddb9c5237%7C0%7C1%7C637335321057136541&amp;sdata=WAxelrAW%2F07e%2Fckmj3nbZIVAkwH0%2FAyJFOoG%2BLf7p14%3D&amp;reserved=0
    --------------------------------------------------------------------
    This email is from an external source. Please do not open attachments or click links from an unknown or suspicious origin. Phishing attempts can be reported by using the report message button in Outlook or sending them as an attachment to [hidden email]. Thank you
    --------------------------------------------------------------------


Information in this email including any attachments may be privileged, confidential and is intended exclusively for the addressee. The views expressed may not be official policy, but the personal views of the originator. If you have received it in error, please notify the sender by return e-mail and delete it from your system. You should not reproduce, distribute, store, retransmit, use or disclose its contents to anyone. Please note we reserve the right to monitor all e-mail communication through our internal and external networks. SKY and the SKY marks are trademarks of Sky Limited and Sky International AG and are used under licence.

Sky UK Limited (Registration No. 2906991), Sky-In-Home Service Limited (Registration No. 2067075), Sky Subscribers Services Limited (Registration No. 2340150) and Sky CP Limited (Registration No. 9513259) are direct or indirect subsidiaries of Sky Limited (Registration No. 2247735). All of the companies mentioned in this paragraph are incorporated in England and Wales and share the same registered office at Grant Way, Isleworth, Middlesex TW7 5QD

-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html