Ways to simplify configs?

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Ways to simplify configs?

Dan Mahoney, System Admin
All,

Is there some way to condense the config into a few files?  I understand
that we want huntgroups and users to be their own things, but my
/usr/local/etc/raddb is currently 45 dirs and 215 files.

Try putting that in puppet, to push out to several servers.  I'm willing
to keep the full tree on my puppet server for editing, but setting that up
for deployment is insane.

Bind, for example, has a tool that will read in a config and output the
relevant, well-formatted, includes-processed, bit.

Any ideas?

-Dan

--

"Don't try to out-wierd me.  I get stranger things than you free with my
breakfast cereal."

-Button seen at I-CON XVII (and subsequently purchased)

--------Dan Mahoney--------
Techie,  Sysadmin,  WebGeek
Gushi on efnet/undernet IRC
ICQ: 13735144   AIM: LarpGM
Site:  http://www.gushi.org
---------------------------

-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Ways to simplify configs?

Jonathan Gazeley
Hi Dan,

Seconded, FreeRADIUS is a complex beast (by necessity). I wrote a Puppet
module that manages a full FreeRADIUS installation, configures modules,
proxying etc. It manages most parts are Puppet resources but the
extra-complex bits (e.g. virtual servers) can be deployed from Puppet as
flat files. You should only need a couple of those. It's pretty
full-featured but there are one or two omissions that I can add if
people need them.

https://forge.puppet.com/jgazeley/freeradius

Let me know how you get on.

Cheers,
Jonathan

--
Jonathan Gazeley
Senior Systems Administrator
IT Services
University of Bristol

On 27/07/17 11:43, Dan Mahoney, System Admin wrote:

> All,
>
> Is there some way to condense the config into a few files?  I
> understand that we want huntgroups and users to be their own things,
> but my /usr/local/etc/raddb is currently 45 dirs and 215 files.
>
> Try putting that in puppet, to push out to several servers.  I'm
> willing to keep the full tree on my puppet server for editing, but
> setting that up for deployment is insane.
>
> Bind, for example, has a tool that will read in a config and output
> the relevant, well-formatted, includes-processed, bit.
>
> Any ideas?
>
> -Dan
>

-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Ways to simplify configs?

Alan DeKok-2
In reply to this post by Dan Mahoney, System Admin
On Jul 27, 2017, at 6:43 AM, Dan Mahoney, System Admin <[hidden email]> wrote:
> Is there some way to condense the config into a few files?  I understand that we want huntgroups and users to be their own things, but my /usr/local/etc/raddb is currently 45 dirs and 215 files.

  You can put almost everything into one file.  The $INCLUDE means "include in place", so you can just do tat with an editor.

  But... your local config probably doesn't actually *use* all of those 45 dirs && 215 files.

> Try putting that in puppet, to push out to several servers.  I'm willing to keep the full tree on my puppet server for editing, but setting that up for deployment is insane.

  Puppet can't push out subdirectories, recursively?  That seems more of a flaw in puppet than anything else.  Unix systems have had subdirectories since 1975 or so...

> Bind, for example, has a tool that will read in a config and output the relevant, well-formatted, includes-processed, bit.

  We're happy to have someone contribute such a tool.

  Alan DeKok.


-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Ways to simplify configs?

Jonathan Gazeley
On 27/07/17 12:46, Alan DeKok wrote:
>> Try putting that in puppet, to push out to several servers.  I'm willing to keep the full tree on my puppet server for editing, but setting that up for deployment is insane.
>    Puppet can't push out subdirectories, recursively?  That seems more of a flaw in puppet than anything else.  Unix systems have had subdirectories since 1975 or so...
>

Puppet *can* push out subdirectories, recursively. However it's not
recommended to do it that way. Puppet's strength is modelling everything
as a resource that can be manipulated in order to generate configs,
rather than directly storing the configs itself. Flat files don't really
tie into this.

Hence I wrote my FreeRADIUS Puppet module which allows the user to
define things in a native Puppet way, and the module then magically
generates the config. It's not quite as flexible as editing the plain
text files but it has enough flexibility for 95% of people's needs, and
it still provides the option to feed in arbitrary plain text configs for
the cases where the content can't be modelled in Puppet.

Cheers,
Jonathan

-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: Ways to simplify configs?

adrian.p.smith
We have a different way of managing this.

1. We store almost a complete raddb folder in a GIT repo with an associated automated integration test suite written in Cucumber-JVM and run by gradle. The tests gzip up the config and, with some minor tweaks, applies it to a local copy of FreeRadius which is spun up by the test suite as required along with other stub servers for downstream radius and REST servers. We use a Java Radius client to feed in packets and assert that things happen, in the stubs and the client, as expected.
2. The project is built in Jenkins and, assuming all the tests pass, the gzip file becomes the deployable artefact
3. We use another tool called Ansible (similar to Puppet) that the uploads and uzips to the production servers after first backing up the existing config. FreeRadius is then automatically restarted.





-----Original Message-----
From: Freeradius-Users [mailto:freeradius-users-bounces+adrian.p.smith=[hidden email]] On Behalf Of Jonathan Gazeley
Sent: 27 July 2017 13:07
To: [hidden email]
Subject: Re: Ways to simplify configs?

On 27/07/17 12:46, Alan DeKok wrote:
>> Try putting that in puppet, to push out to several servers.  I'm willing to keep the full tree on my puppet server for editing, but setting that up for deployment is insane.
>    Puppet can't push out subdirectories, recursively?  That seems more of a flaw in puppet than anything else.  Unix systems have had subdirectories since 1975 or so...
>

Puppet *can* push out subdirectories, recursively. However it's not recommended to do it that way. Puppet's strength is modelling everything as a resource that can be manipulated in order to generate configs, rather than directly storing the configs itself. Flat files don't really tie into this.

Hence I wrote my FreeRADIUS Puppet module which allows the user to define things in a native Puppet way, and the module then magically generates the config. It's not quite as flexible as editing the plain text files but it has enough flexibility for 95% of people's needs, and it still provides the option to feed in arbitrary plain text configs for the cases where the content can't be modelled in Puppet.

Cheers,
Jonathan

-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html

-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Ways to simplify configs?

Alan DeKok-2
On Jul 27, 2017, at 8:21 AM, [hidden email] wrote:
>
> We have a different way of managing this.
>
> 1. We store almost a complete raddb folder in a GIT repo with an associated automated integration test suite written in Cucumber-JVM and run by gradle. The tests gzip up the config and, with some minor tweaks, applies it to a local copy of FreeRadius which is spun up by the test suite as required along with other stub servers for downstream radius and REST servers. We use a Java Radius client to feed in packets and assert that things happen, in the stubs and the client, as expected.

  I tend to recommend using git for the configuration.  It helps with so many things...

> 2. The project is built in Jenkins and, assuming all the tests pass, the gzip file becomes the deployable artefact

  That's probably the simplest thing.

> 3. We use another tool called Ansible (similar to Puppet) that the uploads and uzips to the production servers after first backing up the existing config. FreeRadius is then automatically restarted.

  I've used Salt with good success.  But it handles subdirectories. :(  I just create the config I want in a subdirectory in salt, and then tell it to sync the subdirectory to another machine.

  Alan DeKok.


-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Ways to simplify configs?

Alan DeKok-2
In reply to this post by Jonathan Gazeley
On Jul 27, 2017, at 8:06 AM, Jonathan Gazeley <[hidden email]> wrote:
> Puppet *can* push out subdirectories, recursively. However it's not recommended to do it that way. Puppet's strength is modelling everything as a resource that can be manipulated in order to generate configs, rather than directly storing the configs itself. Flat files don't really tie into this.

  Puppet should be able to have recursive dependencies.

  i.e. create a directory hierarchy from a set of config files / templates / variables.  Then, sync the directory hierarchy to a new machine.

> Hence I wrote my FreeRADIUS Puppet module which allows the user to define things in a native Puppet way, and the module then magically generates the config. It's not quite as flexible as editing the plain text files but it has enough flexibility for 95% of people's needs, and it still provides the option to feed in arbitrary plain text configs for the cases where the content can't be modelled in Puppet.

  Sure, that makes sense.

  Alan DeKok.


-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Ways to simplify configs?

DaveH
In reply to this post by adrian.p.smith
We have something similar, except our configs are still in subversion
(moving to git reasonably soon).

All changes on the dev server are checked in, ready to be checked out on
test. The puppet config specifies both the svn repository and the config
version number to deploy to the production servers.

This allows us to make as many changes as we like in svn, but those
changes never go live until we are happy with load testing and
specifically say in puppet 'now deploy version x'.

Dave


On 27/07/17 13:21, [hidden email] wrote:

> We have a different way of managing this.
>
> 1. We store almost a complete raddb folder in a GIT repo with an associated automated integration test suite written in Cucumber-JVM and run by gradle. The tests gzip up the config and, with some minor tweaks, applies it to a local copy of FreeRadius which is spun up by the test suite as required along with other stub servers for downstream radius and REST servers. We use a Java Radius client to feed in packets and assert that things happen, in the stubs and the client, as expected.
> 2. The project is built in Jenkins and, assuming all the tests pass, the gzip file becomes the deployable artefact
> 3. We use another tool called Ansible (similar to Puppet) that the uploads and uzips to the production servers after first backing up the existing config. FreeRadius is then automatically restarted.
>
>
>
>
>
> -----Original Message-----
> From: Freeradius-Users [mailto:freeradius-users-bounces+adrian.p.smith=[hidden email]] On Behalf Of Jonathan Gazeley
> Sent: 27 July 2017 13:07
> To: [hidden email]
> Subject: Re: Ways to simplify configs?
>
> On 27/07/17 12:46, Alan DeKok wrote:
>>> Try putting that in puppet, to push out to several servers.  I'm willing to keep the full tree on my puppet server for editing, but setting that up for deployment is insane.
>>     Puppet can't push out subdirectories, recursively?  That seems more of a flaw in puppet than anything else.  Unix systems have had subdirectories since 1975 or so...
>>
>
> Puppet *can* push out subdirectories, recursively. However it's not recommended to do it that way. Puppet's strength is modelling everything as a resource that can be manipulated in order to generate configs, rather than directly storing the configs itself. Flat files don't really tie into this.
>
> Hence I wrote my FreeRADIUS Puppet module which allows the user to define things in a native Puppet way, and the module then magically generates the config. It's not quite as flexible as editing the plain text files but it has enough flexibility for 95% of people's needs, and it still provides the option to feed in arbitrary plain text configs for the cases where the content can't be modelled in Puppet.
>
> Cheers,
> Jonathan
>
> -
> List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
>
> -
> List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
>
-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Ways to simplify configs?

Phil Mayers
In reply to this post by Dan Mahoney, System Admin
On 27/07/17 11:43, Dan Mahoney, System Admin wrote:
> All,
>
> Is there some way to condense the config into a few files?  I understand
> that we want huntgroups and users to be their own things, but my
> /usr/local/etc/raddb is currently 45 dirs and 215 files.

As others have said, we put the entire repo into a git directory, and
just push it out to servers.

(In fact, we put it in /etc/raddb-ic and use a separate systemd unit to
point to the new config file; this means and update doesn't modify files
we're using, but we can also look at the "pure stock" config for reference)

-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Ways to simplify configs?

Matthew Newton-3
On Thu, 2017-07-27 at 16:20 +0100, Phil Mayers wrote:
> (In fact, we put it in /etc/raddb-ic and use a separate systemd unit
> to point to the new config file; this means and update doesn't modify
> files we're using, but we can also look at the "pure stock" config
> for reference)

+1 to this.

--
Matthew

-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Ways to simplify configs?

Adam Bishop-2
In reply to this post by Phil Mayers
On 27 Jul 2017, at 16:20, Phil Mayers <[hidden email]> wrote:
> As others have said, we put the entire repo into a git directory, and just push it out to servers.

This, and I split the config into more files than default, not less.

I have a file (/etc/raddb/variables) that is not stored in the git tree that contains the host name of the system, and a bunch of values for whether it's a Dev or Production system, vlan id's, policy enablement etc.

I then have split a bunch of files into folders (e.g. /etc/raddb/clients.d/) to allow one config to serve multiple servers like this:

--+ clients.d/
  |
  +--+ DEV/
  |  |
  |  +--> vpn
  |  +--> eduroam
  |  +--> internal_wifi
  |
  +--+ PROD/
     +--> vpn
     +--> eduroam
     +--> internal_wifi

Which is controlled by a single entry in radiusd.conf:

  $INCLUDE clients.d/${environment}/

There's also certs.d, proxy.d, secrets.d that are managed the same way - for secrets.d the hostname is used instead:
  $INCLUDE secrets.d/${hostname}/

Regards,

Adam Bishop

  gpg: E75B 1F92 6407 DFDF 9F1C  BF10 C993 2504 6609 D460

jisc.ac.uk

Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under Company No. 5747339, VAT No. GB 197 0632 86. Jisc’s registered office is: One Castlepark, Tower Hill, Bristol, BS2 0JA. T 0203 697 5800.

Jisc Services Limited is a wholly owned Jisc subsidiary and a company limited by guarantee which is registered in England under company number 2881024, VAT number GB 197 0632 86. The registered office is: One Castle Park, Tower Hill, Bristol BS2 0JA. T 0203 697 5800.  


-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Ways to simplify configs?

Michael Ströder
In reply to this post by Alan DeKok-2
Alan DeKok wrote:
> On Jul 27, 2017, at 8:21 AM, [hidden email] wrote:
>> 3. We use another tool called Ansible (similar to Puppet) that the uploads and uzips
>> to the production servers after first backing up the existing config. FreeRadius is
>> then automatically restarted.
>
> I've used Salt with good success.  But it handles subdirectories. :(  I just create
> the config I want in a subdirectory in salt, and then tell it to sync the subdirectory
> to another machine.

Your config dir might need different user/group ownership and permissions for certain
files within your directory structure. While puppet/ansible/whatever config management
can all of course "handle" subdirectories it's tricky to get ownership/permissions right
while strictly preserving idempotency.

Reason:
In environments with high security requirements you might have file integrity monitoring
running on all systems (with DB directories or similar excluded). So you definitely aim
to avoid every unnecessary write access during config run because a security auditor has
to manually acknowledge every change, even if only the mtime changed. Copying files
recursively and fixing ownership/permissions afterwards is an anti-pattern in such an
environment.

So I tend to let the config management write only very few config files without too much
sophisticated include dir structure.

Ciao, Michael.



-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html

smime.p7s (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Ways to simplify configs?

Phil Mayers
In reply to this post by Adam Bishop-2
On 27/07/17 16:34, Adam Bishop wrote:

> On 27 Jul 2017, at 16:20, Phil Mayers <[hidden email]>
> wrote:
>> As others have said, we put the entire repo into a git directory,
>> and just push it out to servers.
>
> This, and I split the config into more files than default, not less.
>
> I have a file (/etc/raddb/variables) that is not stored in the git
> tree that contains the host name of the system, and a bunch of values
> for whether it's a Dev or Production system, vlan id's, policy
> enablement etc.

We do very similar things.

We actually run our various services (wireless, eduroam IdP, eduroam SP,
macauth, VPN) as separate processes as opposed to one big radius
process, which provides a measure of protection against crash bugs and
the disruption of a full restart or having to toggle into debug mode; as
such, I have huge sections of our config in "common" includes, and we
use a setup like:

instance.conf

serviceopts {
   name = instance

   # port for nagios to check this
   # instance over authentication
   nagios_port = 16000

   # ditto via status-server
   status_port = 16001

   sql_socks = 10
   redis_socks = 20
   # etc. etc.
}

# global user-supplied
$INCLUDE global.conf
# server-local, not in git
$INCLUDE local.conf
# most of normal radiusd.conf
$INCLUDE common.conf

$INCLUDE sites-enabled/instance
$INCLUDE sites-enabled/instance-inner
$INCLUDE sites-enabled/common-radmin
$INCLUDE sites-enabled/common-status

...and the various included files will make use of ${serviceopts.blah}
expansions to hook all this in.

Works great. Very happy with it.
-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Loading...