12:33 p.m.
ok, so I figure I will see if I can't get some discussion going on
implementation plans. Here's a straw man proposal for everyone to knock
holes in and propose better ways to do our implementation. ;)
* Our deliverable will be a iso image similar to the current netinstall
iso image. It will be that + all packages needed to install/deploy
our featured roles and base things they need. This will allow
installing from the net or using the iso stand alone to install
server roles.
* We add comps groups for each featured server role we support. So, for
example: 'server-freeipa'. This comps group has all the packages
listed we want to install to setup the role as we determine is best.
This may include packages that are not direct dependencies on the
server pacakges.
* The roles use cockpit to manage and configure the server role post
install. If there's a way (not sure if there is), we could look at
adding a anaconda 'spoke' for the server install that runs this at
install time for folks that want to configure then? Or on firstboot.
* We create a new fedora-server-$role package for each role. This role
package contains the 'glue' we add to turn this into a fedora server
role from just a install with the packages and local config. We will
need to come up with guidelines and what these packages will contain,
but:
1 A way to express what items should be monitored when the role is
installed/running. Perhaps /etc/server/$role/monitoring/ports and
processes and such.
2 A way to express data that should be backed up if you want to
backup the role. Perhaps /etc/server/$role/backups/datadirs and
configdirs or databases.
3. We may want to change systemd presets to indicate things that
should be running/enabled/etc when the role package is installed.
4. Anything else we need to impement an "API" for roles. Are we going
to define this soon?
5. We can tie this package to specific versions of requirements. So,
for example, if there's a new freeipa update and we haven't valided
fedora-server-freeipa with it, the requirement can be on the older
version and the users will not be updated until we push an update to
this package.
6. We setup specific bodhi requirements (like critpath) for server
role packages, requiring specific testing before they can go stable.
7. We may have a fedora-server-base or fedora-server-common package
to contain common/base stuff all roles need/user and make the role
packages require it. We would need to be very careful about updates
of this however.
* Users can install fedora-server-freeipa via yum/dnf after the fact,
or via the iso image/anaconda.
* Users can decide they don't care about our valueadding/glue and
remove fedora-server-freeipa and it will just go back to a machine
with all those packages installed (ie, uninstalling shouldnt
hopefully do anything other than allow them to remove required
packages and ignore our old glue).
ok. Poke holes away. ;)
kevin
1:29 p.m.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 01/27/2014 01:33 PM, Kevin Fenzi wrote:
ok, so I figure I will see if I can't get some discussion going
on
implementation plans. Here's a straw man proposal for everyone to
knock holes in and propose better ways to do our implementation. ;)
* Our deliverable will be a iso image similar to the current
netinstall iso image. It will be that + all packages needed to
install/deploy our featured roles and base things they need. This
will allow installing from the net or using the iso stand alone to
install server roles.
* We add comps groups for each featured server role we support. So,
for example: 'server-freeipa'. This comps group has all the
packages listed we want to install to setup the role as we
determine is best. This may include packages that are not direct
dependencies on the server pacakges.
* The roles use cockpit to manage and configure the server role
post install. If there's a way (not sure if there is), we could
look at adding a anaconda 'spoke' for the server install that runs
this at install time for folks that want to configure then? Or on
firstboot.
* We create a new fedora-server-$role package for each role. This
role package contains the 'glue' we add to turn this into a fedora
server role from just a install with the packages and local config.
We will need to come up with guidelines and what these packages
will contain, but:
Ok, so aside from the monitoring/backup stuff you list below, the
primary purpose of this role package is to drop whatever is needed on
the system in order to produce the API that Cockpit will consume to
set up the configuration? (Or, in the more rare cases where a
reasonable default exists, drop that default on the system and go)?
1 A way to express what items should be monitored when the role is
installed/running. Perhaps /etc/server/$role/monitoring/ports and
processes and such.
We really need to come up with a clearer definition of "monitor" here.
What exactly does /etc/server/$role/monitoring/ports mean? Just a set
of ports that have to be accessible publicly for it to be considered
working? I assume we'd also be including firewalld configuration as
part of the role API so that we can handle that properly.
2 A way to express data that should be backed up if you want to
backup the role. Perhaps /etc/server/$role/backups/datadirs and
configdirs or databases.
3. We may want to change systemd presets to indicate things that
should be running/enabled/etc when the role package is installed.
As mentioned above, this should generally be done when Cockpit (or
other management tool) *deploys* the role. It should only be done by
the package if it meets the "may be packaged in such a way that it can
be uninstalled as a complete unit" Conditional Requirement[1]
4. Anything else we need to impement an "API" for roles.
Are we
going to define this soon?
Yes, we need to have a breakout session somewhere to start working
this out. My suggestion would be for us to standardize on something
that is already available on the system (to avoid growing new
dependencies). My suggestion would be for us to build a D-BUS service,
since that will be easily consumed by Cockpit, OpenLMI and many languages.
5. We can tie this package to specific versions of requirements.
So, for example, if there's a new freeipa update and we haven't
valided fedora-server-freeipa with it, the requirement can be on
the older version and the users will not be updated until we push
an update to this package.
I'm going to suggest that *can* has to be *must* (within a certain
reasonable level). I.e. we should probably allow the release number to
change without forcing the role package to be regenerated.
E.g.
Requires mydep >= 1.2.2
Conflicts mydep >= 1.3
as opposed to:
Requires mydep == 1.2.2-1
6. We setup specific bodhi requirements (like critpath) for server
role packages, requiring specific testing before they can go
stable.
I'd *really* like for us to be testing things holistically, rather
than individually. It's probably not enough for us to just check "Is
this in a role package? Yes: require more testing".
I think it's fine for the individual *packages* to continue in the
process that they are now. However, in order to bring something into
the *role*, we should have to test it as a complete system.
This has impact on Fedora Infrastructure, as it will mean that we will
need to retain multiple (non-parallel-installable) versions of
packages in the repositories, since we will always need to be able to
satisfy the role requirements (which may be older than the upstream
package).
Now, I'd be in favor of implementing something like the CRITPATH
process (possibly even the old style, requiring a proventester) for
updating the role package alone. There are two reasons for this:
1) Having the role package installed is a pretty clear indication that
you are actually trying to use the role holistically.
2) It encourages a more comprehensive set of system tests as opposed
to package tests (since people will be testing how the role works,
specifically.
7. We may have a fedora-server-base or fedora-server-common
package to contain common/base stuff all roles need/user and make
the role packages require it. We would need to be very careful
about updates of this however.
* Users can install fedora-server-freeipa via yum/dnf after the
fact, or via the iso image/anaconda.
* Users can decide they don't care about our valueadding/glue and
remove fedora-server-freeipa and it will just go back to a machine
with all those packages installed (ie, uninstalling shouldnt
hopefully do anything other than allow them to remove required
packages and ignore our old glue).
It's off-topic, but I would like to see us also build a
fedora-server-release package that does much the same: allows us to
identify the minimum set of stuff that we call "Fedora Server" and if
you want to remove some pieces of it, you can... but you're no longer
building atop our tested platform.
ok. Poke holes away. ;)
kevin
[1]
https://fedoraproject.org/wiki/Server/Product_Requirements_Document#Condi...
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEARECAAYFAlLms34ACgkQeiVVYja6o6NmzACgqAC/lreABLsGeP1iEUX4KOdW
7nYAnA0Ng54vK3w6CatJBthmpz7lvbPh
=j2BK
-----END PGP SIGNATURE-----
4:41 p.m.
On Mon, 27 Jan 2014 14:29:02 -0500
Stephen Gallagher <sgallagh(a)redhat.com> wrote:
Ok, so aside from the monitoring/backup stuff you list below, the
primary purpose of this role package is to drop whatever is needed on
the system in order to produce the API that Cockpit will consume to
set up the configuration? (Or, in the more rare cases where a
reasonable default exists, drop that default on the system and go)?
Yeah. As well as making sure all the parts needed for the role are
installed.
> 1 A way to express what items should be monitored when the role
is
> installed/running. Perhaps /etc/server/$role/monitoring/ports and
> processes and such.
We really need to come up with a clearer definition of "monitor" here.
What exactly does /etc/server/$role/monitoring/ports mean? Just a set
of ports that have to be accessible publicly for it to be considered
working? I assume we'd also be including firewalld configuration as
part of the role API so that we can handle that properly.
Yeah. There may be some upstream work somewhere we could reuse here.
Some kind of generic monitoring spec that monitoring solutions could
consume? Anyone know of anything in this space?
> 2 A way to express data that should be backed up if you want to
> backup the role. Perhaps /etc/server/$role/backups/datadirs and
> configdirs or databases.
>
> 3. We may want to change systemd presets to indicate things that
> should be running/enabled/etc when the role package is installed.
>
As mentioned above, this should generally be done when Cockpit (or
other management tool) *deploys* the role. It should only be done by
the package if it meets the "may be packaged in such a way that it can
be uninstalled as a complete unit" Conditional Requirement[1]
Sure. There's going to be some cases of things you don't want to start
on boot until configured, etc.
> 4. Anything else we need to impement an "API" for
roles. Are we
> going to define this soon?
Yes, we need to have a breakout session somewhere to start working
this out. My suggestion would be for us to standardize on something
that is already available on the system (to avoid growing new
dependencies). My suggestion would be for us to build a D-BUS service,
since that will be easily consumed by Cockpit, OpenLMI and many
languages.
So, this would be a thing that looked at all the config put down in
the role rpm and published it and changes over dbus for any consumers?
We are writing this? Or is there something generic we can reuse with
our needed config?
> 5. We can tie this package to specific versions of
requirements.
> So, for example, if there's a new freeipa update and we haven't
> valided fedora-server-freeipa with it, the requirement can be on
> the older version and the users will not be updated until we push
> an update to this package.
>
I'm going to suggest that *can* has to be *must* (within a certain
reasonable level). I.e. we should probably allow the release number to
change without forcing the role package to be regenerated.
E.g.
Requires mydep >= 1.2.2
Conflicts mydep >= 1.3
as opposed to:
Requires mydep == 1.2.2-1
Sure, we could do that. The goal there would be to stop someone from
accidentally upgrading past our tested versions.
> 6. We setup specific bodhi requirements (like critpath) for
server
> role packages, requiring specific testing before they can go
> stable.
>
I'd *really* like for us to be testing things holistically, rather
than individually. It's probably not enough for us to just check "Is
this in a role package? Yes: require more testing".
Sure, I think we should do that _also_
This would just be a way to catch things like "hey, freeipa is bumping
a major release in a stable version" (just an example)
I think it's fine for the individual *packages* to continue in
the
process that they are now. However, in order to bring something into
the *role*, we should have to test it as a complete system.
Sure, since the role package requires specific things tho, doesn't
testing it also mean testing those?
This has impact on Fedora Infrastructure, as it will mean that we
will
need to retain multiple (non-parallel-installable) versions of
packages in the repositories, since we will always need to be able to
satisfy the role requirements (which may be older than the upstream
package).
This is difficult for a number of reasons. :) (Not impossible, but
difficult).
However, we should always have the initial version at release.
And we should be able to stop a new version in testing that breaks
things/is a major version bump, no? We don't want to be pushing major
updates into stable releases...
Now, I'd be in favor of implementing something like the CRITPATH
process (possibly even the old style, requiring a proventester) for
updating the role package alone. There are two reasons for this:
1) Having the role package installed is a pretty clear indication that
you are actually trying to use the role holistically.
2) It encourages a more comprehensive set of system tests as opposed
to package tests (since people will be testing how the role works,
specifically.
Sure.
...snip...
It's off-topic, but I would like to see us also build a
fedora-server-release package that does much the same: allows us to
identify the minimum set of stuff that we call "Fedora Server" and if
you want to remove some pieces of it, you can... but you're no longer
building atop our tested platform.
Yeah, might be worth doing...
kevin
7:43 a.m.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 01/27/2014 05:41 PM, Kevin Fenzi wrote:
<snip>
On Mon, 27 Jan 2014 14:29:02 -0500 Stephen Gallagher
<sgallagh(a)redhat.com> wrote:
>> 4. Anything else we need to impement an "API" for roles. Are
>> we going to define this soon?
>
> Yes, we need to have a breakout session somewhere to start
> working this out. My suggestion would be for us to standardize on
> something that is already available on the system (to avoid
> growing new dependencies). My suggestion would be for us to build
> a D-BUS service, since that will be easily consumed by Cockpit,
> OpenLMI and many languages.
So, this would be a thing that looked at all the config put down
in the role rpm and published it and changes over dbus for any
consumers? We are writing this? Or is there something generic we
can reuse with our needed config?
This is likely something we're going to need to write ourselves (with
an eye to getting other major distributions involved, since it will
benefit them as well).
My thought here is that what we'd probably want to do is build a
configuration service that serves a D-BUS API. The service should be
built around a plugin architecture, so that we can drop a new set of
objects onto it as part of the role package. Ideally, I'd like these
modules to be built atop something like Augeas so that we aren't
responsible for actually managing the files on-disk, just presenting
the data in the API.
I'm aware that this is non-trivial new work. If we agree that this is
what we want and need, I'm going to make an attempt at requisitioning
resources to work on it.
<snip>
>> 6. We setup specific bodhi requirements (like critpath) for
>> server role packages, requiring specific testing before they
>> can go stable.
>>
>
> I'd *really* like for us to be testing things holistically,
> rather than individually. It's probably not enough for us to just
> check "Is this in a role package? Yes: require more testing".
Sure, I think we should do that _also_
This would just be a way to catch things like "hey, freeipa is
bumping a major release in a stable version" (just an example)
I'm not sure I agree with the need for additional process here, but I
am open to being convinced. For now, I'd prefer to play the
wait-and-see game and only add to this process if we see it becoming
an issue.
> I think it's fine for the individual *packages* to continue
in
> the process that they are now. However, in order to bring
> something into the *role*, we should have to test it as a
> complete system.
Sure, since the role package requires specific things tho, doesn't
testing it also mean testing those?
No. In many cases it only means testing a specific subset of those
dependencies. For example, FreeIPA depends on 389 DS and Dogtag, so by
your logic, testing FreeIPA is sufficient to determine that newer 389
DS and Dogtag packages work. However, there are numerous features
(particularly in Dogtag) that FreeIPA does not exercise. So I see
value in testing the packages separately from the roles.
> This has impact on Fedora Infrastructure, as it will mean that
we
> will need to retain multiple (non-parallel-installable) versions
> of packages in the repositories, since we will always need to be
> able to satisfy the role requirements (which may be older than
> the upstream package).
This is difficult for a number of reasons. :) (Not impossible, but
difficult).
Could you explain some of those reasons? To me it seems like we'd be
moving from maintaining two copies (the one in the release repo plus
the one in the updates repo) with a possibility of three or more (one
for each role that requires a unique version compared to the updates
and other role requirements).
Certainly, we should strive to ensure that this number remains 2 as
often as possible.
However, we should always have the initial version at release. And
we should be able to stop a new version in testing that breaks
things/is a major version bump, no? We don't want to be pushing
major updates into stable releases...
It's always difficult to determine what is a major update vs. a
compatible update that adds a few new features. I think the goal here
would be to allow compatible updates to land with the expectation that
we're going to test them with the roles and update the role packages
as soon as it's verified.
<snip>
> It's off-topic, but I would like to see us also build a
> fedora-server-release package that does much the same: allows us
> to identify the minimum set of stuff that we call "Fedora Server"
> and if you want to remove some pieces of it, you can... but
> you're no longer building atop our tested platform.
Yeah, might be worth doing...
The other side-effect here would be to make it easy for Cloud to
implement its Cloud->Server goal (just install the
fedora-server-release package and your system now meets that condition).
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEARECAAYFAlLns+8ACgkQeiVVYja6o6McPwCgpZR/BtfuF+F+J789yWiE9BmL
K04AoJyFSHk3+ekLFg6W/DGoC/Nd+FFb
=1trv
-----END PGP SIGNATURE-----
11:46 a.m.
On Tue, 28 Jan 2014 08:43:11 -0500
Stephen Gallagher <sgallagh(a)redhat.com> wrote:
This is likely something we're going to need to write ourselves
(with
an eye to getting other major distributions involved, since it will
benefit them as well).
My thought here is that what we'd probably want to do is build a
configuration service that serves a D-BUS API. The service should be
built around a plugin architecture, so that we can drop a new set of
objects onto it as part of the role package. Ideally, I'd like these
modules to be built atop something like Augeas so that we aren't
responsible for actually managing the files on-disk, just presenting
the data in the API.
I'm aware that this is non-trivial new work. If we agree that this is
what we want and need, I'm going to make an attempt at requisitioning
resources to work on it.
Yeah, it sounds like it could be a lot of work, but it could also be
potentially a very good value add.
In the end we always need to keep in mind: why would someone install
the fedora server freeipa role instead of just installing the packages
and configuring it themselves as they do now? We need to make it
compelling to use the role. :)
>>> 6. We setup specific bodhi requirements (like critpath)
for
>>> server role packages, requiring specific testing before they
>>> can go stable.
>>>
>>
>> I'd *really* like for us to be testing things holistically,
>> rather than individually. It's probably not enough for us to just
>> check "Is this in a role package? Yes: require more testing".
>
> Sure, I think we should do that _also_
>
> This would just be a way to catch things like "hey, freeipa is
> bumping a major release in a stable version" (just an example)
>
I'm not sure I agree with the need for additional process here, but I
am open to being convinced. For now, I'd prefer to play the
wait-and-see game and only add to this process if we see it becoming
an issue.
Sure, this could definitely be something we push off for now, but
adjust as we get closer to some kind of alpha release.
>> I think it's fine for the individual *packages* to
continue in
>> the process that they are now. However, in order to bring
>> something into the *role*, we should have to test it as a
>> complete system.
>
> Sure, since the role package requires specific things tho, doesn't
> testing it also mean testing those?
>
No. In many cases it only means testing a specific subset of those
dependencies. For example, FreeIPA depends on 389 DS and Dogtag, so by
your logic, testing FreeIPA is sufficient to determine that newer 389
DS and Dogtag packages work. However, there are numerous features
(particularly in Dogtag) that FreeIPA does not exercise. So I see
value in testing the packages separately from the roles.
Sure, again, I think both are good.
>> This has impact on Fedora Infrastructure, as it will mean
that we
>> will need to retain multiple (non-parallel-installable) versions
>> of packages in the repositories, since we will always need to be
>> able to satisfy the role requirements (which may be older than
>> the upstream package).
>
> This is difficult for a number of reasons. :) (Not impossible, but
> difficult).
>
Could you explain some of those reasons? To me it seems like we'd be
moving from maintaining two copies (the one in the release repo plus
the one in the updates repo) with a possibility of three or more (one
for each role that requires a unique version compared to the updates
and other role requirements).
So, the way composes work now is that bodhi calls mash and mash calls
out to koji and says "hey, give me the latest package tagged into this
tag" and then it composes say 'fedora/20/updates' repo from that.
If you wanted to keep more than the latest, we would have to call out
for 'give me all packages in the tag, and then figure out which are
the latest two (or whatever) and compose that'. Which is a great deal
more intensive and error prone. You have to think of things like epoch
changes, etc.
Getting past that, the other big hurdle is mirrors. "Hey, you know
that 1TB of fedora you have? We are going to make it 2 or 3, hope
thats ok"
Getting back to the orig statement, why wouldn't we just not push a
version to stable until it meets the requirements of the role?
It's always difficult to determine what is a major update vs. a
compatible update that adds a few new features. I think the goal here
would be to allow compatible updates to land with the expectation that
we're going to test them with the roles and update the role packages
as soon as it's verified.
We should be able to use updates-testing for this, no?
new version enters updates-testing -> we test with role, update role
and push it's new version into updates-testing (preferably with the
same update as the new package so they go stable at the same time).
...snip...
The other side-effect here would be to make it easy for Cloud to
implement its Cloud->Server goal (just install the
fedora-server-release package and your system now meets that
condition).
Yep.
kevin
8:06 p.m.
On Mon, Jan 27, 2014 at 11:33:30AM -0700, Kevin Fenzi wrote:
* The roles use cockpit to manage and configure the server role post
install. If there's a way (not sure if there is), we could look at
adding a anaconda 'spoke' for the server install that runs this at
install time for folks that want to configure then? Or on firstboot.
As I understand it, yes, that could be a thing. Possibly even a new "hub",
with spokes for each selected server role, if chosing more than one is a
supported configuration. Bonus: the same code can be used in firstboot, so
you can easily develop both or either options.
More here:
http://vpodzime.fedorapeople.org/anaconda-addon-development-guide
--
Matthew Miller -- Fedora Project -- <mattdm(a)fedoraproject.org>
8:28 p.m.
On January 27, 2014 09:06:31 PM Matthew Miller wrote:
On Mon, Jan 27, 2014 at 11:33:30AM -0700, Kevin Fenzi wrote:
> * The roles use cockpit to manage and configure the server role post
>
> install. If there's a way (not sure if there is), we could look at
> adding a anaconda 'spoke' for the server install that runs this at
> install time for folks that want to configure then? Or on firstboot.
As I understand it, yes, that could be a thing. Possibly even a new "hub",
with spokes for each selected server role, if chosing more than one is a
supported configuration. Bonus: the same code can be used in firstboot, so
you can easily develop both or either options.
More here:
http://vpodzime.fedorapeople.org/anaconda-addon-development-guide
Hello folks,
I previously worked with Red Hat on their Red Hat HPC Solution (our original Fork/redo of
NPACI Rocks which was.. difficult). It's discontinued now.
We had the concept of kits to do metapackages. I have the last CVS snapshot code available
for people to examine. I will have to mirror it somewhere as my laptop as a VM/web server
can only handle so much bandwidth :)
The concept was this:
kit-something ---> pulls in: kit-client-configs / kit-server-configs <--- pulls in
RPM
dependencies for packages. Kits == Roles basically.
Now, we didn't have comps.xml to work with but we could maybe do something like: We
have a roles.xml which maps to comps.xml groups.
So, then "Fedora Server Role" maps to X groups in comps.xml which pulls in the
direct RPM
dependencies.
The "Fedora Server Role" RPM/whatever can have then its own "turn key"
configuration
template/setup files for the groups it pulls in to configure them out-of-the-box.
so:
role-something ---> pulls in: role-client-configs / role-server-configs <--- pulls
in comps.xml
group package RPM dependencies as part of each grouping
That way, we dont break existing users where a Desktop also contains server configurations
(comps.xml -> RPM dependencies only) and we gain the ability to have a higher-level
grouping which adds on turn-key meta configurations that are *optional*.
So its a one way-dependency mapping and both use cases win!
Thoughts?
Thanks,
Shawn
11:50 a.m.
On Mon, 27 Jan 2014 21:28:37 -0500
Shawn Starr <shawn.starr(a)rogers.com> wrote:
Hello folks,
Hey.
I previously worked with Red Hat on their Red Hat HPC Solution (our
original Fork/redo of NPACI Rocks which was.. difficult). It's
discontinued now.
We had the concept of kits to do metapackages. I have the last CVS
snapshot code available for people to examine. I will have to mirror
it somewhere as my laptop as a VM/web server can only handle so much
bandwidth :)
The concept was this:
kit-something ---> pulls in: kit-client-configs / kit-server-configs
<--- pulls in RPM dependencies for packages. Kits == Roles basically.
Now, we didn't have comps.xml to work with but we could maybe do
something like: We have a roles.xml which maps to comps.xml groups.
So, then "Fedora Server Role" maps to X groups in comps.xml which
pulls in the direct RPM dependencies.
The "Fedora Server Role" RPM/whatever can have then its own "turn
key" configuration template/setup files for the groups it pulls in to
configure them out-of-the-box.
so:
role-something ---> pulls in: role-client-configs /
role-server-configs <--- pulls in comps.xml group package RPM
dependencies as part of each grouping
That way, we dont break existing users where a Desktop also contains
server configurations (comps.xml -> RPM dependencies only) and we
gain the ability to have a higher-level grouping which adds on
turn-key meta configurations that are *optional*.
So its a one way-dependency mapping and both use cases win!
Thoughts?
metapackages have a number of issues... see the cons section of:
http://yum.baseurl.org/wiki/YumGroupsOverview
In the case of just a 'fedora-server-freeipa' role, they make sense to
me, as if someone removes a specific package that this role needs, it
would remove the role package, which is fine, since they would no
longer be using the role.
I'm not sure what deeper level you are suggesting here, can you provide
a more detailed example?
kevin
6:50 p.m.
On January 29, 2014 10:50:32 AM Kevin Fenzi wrote:
On Mon, 27 Jan 2014 21:28:37 -0500
Shawn Starr <shawn.starr(a)rogers.com> wrote:
> Hello folks,
Hey.
> I previously worked with Red Hat on their Red Hat HPC Solution (our
> original Fork/redo of NPACI Rocks which was.. difficult). It's
> discontinued now.
>
> We had the concept of kits to do metapackages. I have the last CVS
> snapshot code available for people to examine. I will have to mirror
> it somewhere as my laptop as a VM/web server can only handle so much
> bandwidth :)
>
> The concept was this:
>
> kit-something ---> pulls in: kit-client-configs / kit-server-configs
> <--- pulls in RPM dependencies for packages. Kits == Roles basically.
>
> Now, we didn't have comps.xml to work with but we could maybe do
> something like: We have a roles.xml which maps to comps.xml groups.
>
> So, then "Fedora Server Role" maps to X groups in comps.xml which
> pulls in the direct RPM dependencies.
>
> The "Fedora Server Role" RPM/whatever can have then its own "turn
> key" configuration template/setup files for the groups it pulls in to
> configure them out-of-the-box.
>
> so:
>
> role-something ---> pulls in: role-client-configs /
> role-server-configs <--- pulls in comps.xml group package RPM
> dependencies as part of each grouping
>
> That way, we dont break existing users where a Desktop also contains
> server configurations (comps.xml -> RPM dependencies only) and we
> gain the ability to have a higher-level grouping which adds on
> turn-key meta configurations that are *optional*.
>
> So its a one way-dependency mapping and both use cases win!
>
> Thoughts?
metapackages have a number of issues... see the cons section of:
http://yum.baseurl.org/wiki/YumGroupsOverview
In the case of just a 'fedora-server-freeipa' role, they make sense to
me, as if someone removes a specific package that this role needs, it
would remove the role package, which is fine, since they would no
longer be using the role.
I'm not sure what deeper level you are suggesting here, can you provide
a more detailed example?
kevin
Hi Kevin,
I've uploaded the original Red Hat HPC Solution source (the last snapshot of code I
had)to Fedora People, you can browse it here:
http://spstarr.fedorapeople.org/ocs-rhhpc/trunk/src/
The kits concept is shown in the .spec files and how we did plugins / meta
configurations.
Thanks,
Shawn
12:12 a.m.
Fedora Server Folks,
Users can decide they don't care about our valueadding/glue and
remove fedora-server-freeipa and it will just go back to a machine
with all those packages installed (ie, uninstalling shouldnt
hopefully do anything other than allow them to remove required
packages and ignore our old glue).
I'm hoping that the building blocks are modular enough that users can put together
their own server roles. I had not been following the group recently as I thought our
visions had diverged (and I'm stuck in the EL6 world) but it appears that we are in
fact very close.
I do our "server roles" in RPMs today. I do hope the Fedora roles are modular
so I personally don't have to maintain RPMs for common things like installing a
running web server. I've done this for our RPMs by simply adding "-on" RPMs
for example postgresql-server-on, httpd-on, etc.
Here are some of the packages that I could contribute but since I'm not and expert
they probably have issues or are trivial in the grand scheme of things. But, these
packages have saved us thousand of hours over the long term.
chrony-on
couchdb-on
crond-on
httpd-on
mysql-server-on
nagios-on
ntpd-on
openssh-server-on
postgis-on
postgresql-server-on
A package I just wrote is the "don't install documentation on a headless server
package". I was surprised that I could not find anything like it on the net. It
saves a lot of drive space on servers where you just don't need HTML or man pages.
Thanks,
Mike
mrdvt92
11:52 a.m.
On Mon, 27 Jan 2014 22:12:58 -0800 (PST)
"Michael R. Davis" <mrdvt92(a)yahoo.com> wrote:
I'm hoping that the building blocks are modular enough that users
can
put together their own server roles.
I sure hope so, this is the only way we are going to grow roles over
time. :)
I had not been following the
group recently as I thought our visions had diverged (and I'm stuck
in the EL6 world) but it appears that we are in fact very close. I do
our "server roles" in RPMs today. I do hope the Fedora roles are
modular so I personally don't have to maintain RPMs for common
things like installing a running web server. I've done this for our
RPMs by simply adding "-on" RPMs for example postgresql-server-on,
httpd-on, etc. Here are some of the packages that I could contribute
but since I'm not and expert they probably have issues or are trivial
in the grand scheme of things. But, these packages have saved us
thousand of hours over the long term. chrony-on couchdb-on crond-on
httpd-on mysql-server-on nagios-on ntpd-on
openssh-server-on
postgis-on
postgresql-server-on
I don't see a need for 'on' packages, but a role could ship a systemd
preset or config that would enable services needed as part of that
config.
A package I just wrote is the "don't install documentation
on a
headless server package". I was surprised that I could not find
anything like it on the net. It saves a lot of drive space on
servers where you just don't need HTML or man pages. Thanks, Mike
What does this package contain out of curiosity?
kevin
1:37 p.m.
From: Kevin Fenzi <kevin(a)scrye.com>
Sent: Wednesday, January 29, 2014 12:52 PM
>
> On Mon, 27 Jan 2014 22:12:58 -0800 (PST)
> "Michael R. Davis" <mrdvt92(a)yahoo.com> wrote:
>
> I've done this for our
> RPMs by simply adding "-on" RPMs for example postgresql-server-on,
> httpd-on, etc. Here are some of the packages that I could contribute
> but since I'm not and expert they probably have issues or are trivial
> in the grand scheme of things. But, these packages have saved us
> thousand of hours over the long term. chrony-on couchdb-on crond-on
> httpd-on mysql-server-on nagios-on ntpd-on
> openssh-server-on
> postgis-on
> postgresql-server-on
I don't see a need for 'on' packages, but a role could ship a systemd
preset or config that would enable services needed as part of that
config.
I'm not sure I care how roles gets a running httpd server but, it's a common task
that 1000s of people need to do everyday.
In the spec "Require: httpd-on" is a whole lot easier than doing something in a
%post section
%post
do_what_needs_to_be_done_to_get_httpd_running_and_open_port_80_on_the_firewall
I guess bottom line it is currently way too hard to deploy applications via RPMs as
there's no foundation. We just need to build the foundations so Apps can be as easy
as
Require: httpd-on
Require: postgresql-server-on
Require: perl(App::Whatever)
%files
/etc/httpd/conf.d/App-Whatever.conf
> A package I just wrote is the "don't install
documentation on a
> headless server package". I was surprised that I could not find
> anything like it on the net. It saves a lot of drive space on
> servers where you just don't need HTML or man pages.
What does this package contain out of curiosity?
Excerpt...
$ cat rpm-rpmmacros-server.rpmmacros
%_excludedocs 1
$ cat rpm-rpmmacros-server.spec
%define myrootdir /root
cp rpm-rpmmacros-server.rpmmacros $RPM_BUILD_ROOT/%{myrootdir}/.rpmmacros
Please remember we are a small business about 100 people with only three Linux people
(none of us are experts). A big company can afford to re-invent the wheel. We cannot.
Thanks,
Mike
mrdvt92
2:08 p.m.
BTW, I am on the server list, there's no need to also cc me. ;)
On Wed, 29 Jan 2014 11:37:35 -0800 (PST)
"Michael R. Davis" <mrdvt92(a)yahoo.com> wrote:
I'm not sure I care how roles gets a running httpd server but, it's a
common task that 1000s of people need to do everyday.
Sure, but before you run it you need some basic configuration no?
Or at least for it to be useful...
In the spec "Require: httpd-on" is a whole lot easier than
doing
something in a %post section
%post
do_what_needs_to_be_done_to_get_httpd_running_and_open_port_80_on_the_firewall
I wouldn't do that in a post no.
I would let the thing that you use to configure it do that.
send a 'httpd is configured' dbus message, then that starts and enables
the service and opens the port(s)
...snip...
>> A package I just wrote is the "don't install
documentation on a
>> headless server package". I was surprised that I could not find
>> anything like it on the net. It saves a lot of drive space on
>> servers where you just don't need HTML or man pages.
>
> What does this package contain out of curiosity?
>
Excerpt...
$ cat rpm-rpmmacros-server.rpmmacros
%_excludedocs 1
$ cat rpm-rpmmacros-server.spec
%define myrootdir /root
cp rpm-rpmmacros-server.rpmmacros
$RPM_BUILD_ROOT/%{myrootdir}/.rpmmacros
Right. There are some issues with that sadly... like you can't install
just the docs later easily, and rpm -V shows them missing, etc.
I wish rpm had better handling for that.
kevin
4:55 a.m.
On 29.01.2014 21:08, Kevin Fenzi wrote:
Right. There are some issues with that sadly... like you can't
install
just the docs later easily, and rpm -V shows them missing, etc.
I wish rpm had better handling for that.
Wishful thinking will get you nowhere. :)
Is this really rpm problem. :)
poma
12:29 p.m.
Hi,
What about discussing implementation after we choose a few roles?
I think FreeIPA has been mentioned before - it would make a good first
featured role, and I think it's also probably something to which one
would reasonably dedicate a machine. (As opposed to e.g. putting it
in a Docker)
10:39 a.m.
On Tue, 11 Feb 2014 18:29:37 +0000
Colin Walters <walters(a)verbum.org> wrote:
Hi,
What about discussing implementation after we choose a few roles?
I think FreeIPA has been mentioned before - it would make a good
first featured role, and I think it's also probably something to
which one would reasonably dedicate a machine. (As opposed to e.g.
putting it in a Docker)
Well, we already decided our first role would be a freeipa server. ;)
I don't know how far we want to go on the first cut, but it might be
worth picking 1 or 2 more so we can also look at how roles interact.
kevin
9:22 p.m.
On 13.02.2014 18:39, Kevin Fenzi wrote:
Well, we already decided our first role would be a freeipa server.
;)
I don't know how far we want to go on the first cut, but it might be
worth picking 1 or 2 more so we can also look at how roles interact.
One item from my (long :-) ) personal wishlist:
Fedora PXE server: In a FreeIPA LAN it is very likely non Fedora OSs to
coexist. It would be nice for some period of testing, education and
migration to Fedora Workstation the MACs of these machines to be
registered for PXE boot with network root (e.g. qemu-nbd or iscsi on
lvm-thinp serving base F21 image + overlay per MAC). Optionally with
included ImageFactory for generating base images for the local spins.
Kind Regards,
Alek
10:29 a.m.
On Thu, 2014-02-13 at 09:39 -0700, Kevin Fenzi wrote:
On Tue, 11 Feb 2014 18:29:37 +0000
Colin Walters <walters(a)verbum.org> wrote:
> Hi,
>
> What about discussing implementation after we choose a few roles?
>
> I think FreeIPA has been mentioned before - it would make a good
> first featured role, and I think it's also probably something to
> which one would reasonably dedicate a machine. (As opposed to e.g.
> putting it in a Docker)
Well, we already decided our first role would be a freeipa server. ;)
I don't know how far we want to go on the first cut, but it might be
worth picking 1 or 2 more so we can also look at how roles interact.
What would be the next thing you instal once you have a netowrk and
identity infrastructure?
To be honest I'd refrain from "fileserver" now because I know how
complicated storage becomes very quickly, so before we tackle such a
role I think we need to work on a few easier ones that do not have the
huge variability a "file server" may have.
Simo.
--
Simo Sorce * Red Hat, Inc * New York
10:43 a.m.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 02/21/2014 11:29 AM, Simo Sorce wrote:
On Thu, 2014-02-13 at 09:39 -0700, Kevin Fenzi wrote:
> On Tue, 11 Feb 2014 18:29:37 +0000 Colin Walters
> <walters(a)verbum.org> wrote:
>
>> Hi,
>>
>> What about discussing implementation after we choose a few
>> roles?
>>
>> I think FreeIPA has been mentioned before - it would make a
>> good first featured role, and I think it's also probably
>> something to which one would reasonably dedicate a machine.
>> (As opposed to e.g. putting it in a Docker)
>
> Well, we already decided our first role would be a freeipa
> server. ;)
>
> I don't know how far we want to go on the first cut, but it might
> be worth picking 1 or 2 more so we can also look at how roles
> interact.
What would be the next thing you instal once you have a netowrk
and identity infrastructure?
To be honest I'd refrain from "fileserver" now because I know how
complicated storage becomes very quickly, so before we tackle such
a role I think we need to work on a few easier ones that do not
have the huge variability a "file server" may have.
So let's take a look at what people use Fedora (or RHEL) servers for
today.
The server platform is there so that people can deploy their own apps
on it. So for us to have value, our Roles should be providing tools
that make these apps simpler. So let's look at commonalities.
Most apps these days need to talk to a database of some sort (be it
traditional like MariaDB or PostgreSQL or the new stuff like MongoDB).
A lot of web applications these days will also use a centralized
memcached host to accelerate their responses.
Naturally, there's also the plain old 'webserver', but that's probably
in a similar level of complexity to "fileserver", so we may want to
hold off on that for now.
I'd suggest that the second Role we might want to offer would be a
standard Database role. We can pick either MariaDB or PostgreSQL (by
majority vote) and run with it. (That's not to say we couldn't offer
both eventually, but let's not split our attention initially).
This should be relatively straightforward since setting up the
databases really requires only configuring authentication and starting
up the service. We can do a little integration work to ensure that the
auth works reasonably well with our identity management system and
then work on a configuration API to simplify creating new databases
and permissions on them. Probably not an insurmountable amount of work.
Thoughts?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEUEARECAAYFAlMHgkoACgkQeiVVYja6o6NYHQCgoH6A4lNFQvVrOKEr678v+m7l
nTsAmPK23Sk+JQJP1IONRRsN7Fei6IM=
=pW+F
-----END PGP SIGNATURE-----
10:55 a.m.
On Fri, 2014-02-21 at 11:43 -0500, Stephen Gallagher wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 02/21/2014 11:29 AM, Simo Sorce wrote:
> On Thu, 2014-02-13 at 09:39 -0700, Kevin Fenzi wrote:
>> On Tue, 11 Feb 2014 18:29:37 +0000 Colin Walters
>> <walters(a)verbum.org> wrote:
>>
>>> Hi,
>>>
>>> What about discussing implementation after we choose a few
>>> roles?
>>>
>>> I think FreeIPA has been mentioned before - it would make a
>>> good first featured role, and I think it's also probably
>>> something to which one would reasonably dedicate a machine.
>>> (As opposed to e.g. putting it in a Docker)
>>
>> Well, we already decided our first role would be a freeipa
>> server. ;)
>>
>> I don't know how far we want to go on the first cut, but it might
>> be worth picking 1 or 2 more so we can also look at how roles
>> interact.
>
> What would be the next thing you instal once you have a netowrk
> and identity infrastructure?
>
> To be honest I'd refrain from "fileserver" now because I know how
> complicated storage becomes very quickly, so before we tackle such
> a role I think we need to work on a few easier ones that do not
> have the huge variability a "file server" may have.
>
So let's take a look at what people use Fedora (or RHEL) servers for
today.
The server platform is there so that people can deploy their own apps
on it. So for us to have value, our Roles should be providing tools
that make these apps simpler. So let's look at commonalities.
Most apps these days need to talk to a database of some sort (be it
traditional like MariaDB or PostgreSQL or the new stuff like MongoDB).
A lot of web applications these days will also use a centralized
memcached host to accelerate their responses.
Naturally, there's also the plain old 'webserver', but that's probably
in a similar level of complexity to "fileserver", so we may want to
hold off on that for now.
I'd suggest that the second Role we might want to offer would be a
standard Database role. We can pick either MariaDB or PostgreSQL (by
majority vote) and run with it. (That's not to say we couldn't offer
both eventually, but let's not split our attention initially).
This should be relatively straightforward since setting up the
databases really requires only configuring authentication and starting
up the service. We can do a little integration work to ensure that the
auth works reasonably well with our identity management system and
then work on a configuration API to simplify creating new databases
and permissions on them. Probably not an insurmountable amount of work.
Thoughts?
I was thinking on similar lines, though I am not a DBA and my experience
with databases is limited, just enough to hate one (multiple data
losses) and be comfortable with the other (just works :)
Do we have anyone on this list that have reasonable experience with
setting up SQL databases ?
Any reason why we should, instead, choose something else ?
Simo.
--
Simo Sorce * Red Hat, Inc * New York
11:34 a.m.
Do we have anyone on this list that have reasonable experience with
setting up SQL databases ?
We have database rpms which provide running database servers for our systems.
postgis-on
postgresql-server-on
mysql-server-on
They are trivial specs that just do simple stuff like...
%post
chkconfig mysqld on
service mysqld start
But, they are invaluable in that it's done the same install on every server in our
network. If they were done well they also have hooks into iptables and whatever else was
needed to secure the system. (this is where roles come in role:intranet vs role:internet)
I'd love to have this capability in the base release. So, I would not have to manage
these myself and as I've stated many times: I am no expert in databases nor security.
Have any of you tried to install wordpress. It makes me sick that they just don't
install a default database configuration right out of the RPMs. It raises the bar too
high for many and puts a lot of burden on the application developers to build nice install
scripts for every case out there.
It could be as easy as...
wordpress.spec
Requires: database(running)
Requires: webserver(running)
%post
wordpress_install_database %database
wordpress_install_webserver %webserver
Thanks,
Mike
12:02 p.m.
On Fri, 2014-02-21 at 09:34 -0800, Michael R. Davis wrote:
Have any of you tried to install wordpress. It makes me sick that
they just don't install a default database configuration right out of
the RPMs.
Think about what we're defining here: a 'database server' role.
The point of having a single database server is that you don't run a
database on every system where you have something installed that needs a
database.
The reason Fedora webapp packages don't deploy or expect a database
running on the same machine is that, frequently, people don't want that.
They want to use a database running on another machine.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
12:12 p.m.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 02/21/2014 01:02 PM, Adam Williamson wrote:
On Fri, 2014-02-21 at 09:34 -0800, Michael R. Davis wrote:
>
> Have any of you tried to install wordpress. It makes me sick
> that they just don't install a default database configuration
> right out of the RPMs.
Think about what we're defining here: a 'database server' role.
The point of having a single database server is that you don't run
a database on every system where you have something installed that
needs a database.
The reason Fedora webapp packages don't deploy or expect a
database running on the same machine is that, frequently, people
don't want that. They want to use a database running on another
machine.
I'd make the strong argument that specific apps like this would be
better served by something like OpenShift cartridges (which can be
configured to use either a local or remote DB server) than as Server
Roles.
Fedora Server having an "OpenShift Node" Server Role is another thing
I want to see us do (essentially having a really easy way to add new
machines to an OpenShift cloud). However, I suspect this will require
significant work and is probably out of scope for our F21 deliverable
(but it's something I'm going to be coordinating with the OpenShift team).
I wonder if our definition of Server Roles was too ambiguous in the
PRD. My view was that it should be a way to set up clear platform
states and certain infrastructure services. I think that deploying
specific "apps" (and I know the term is fuzzy... is FreeIPA an app?)
is not really the purpose of a Role.
So for some off-the-top-of-my-head examples:
Infrastructure: Identity management, database server, OpenShift Node,
OpenStack Node, File Server, Web Server
App: Wordpress, Drupal, SAP
Drupal is a little fuzzy, since it's both an app and something of an
application framework, but I'd elect to punt on that decision for the
indefinite future :)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEARECAAYFAlMHlwwACgkQeiVVYja6o6MZUgCeIFyJPSgdzRThmthgmW9cNvdM
XJ4An1Hfr2EZKlpAm5yBww3yDNwYr4pc
=4j8a
-----END PGP SIGNATURE-----
noon
On Fri, 2014-02-21 at 11:43 -0500, Stephen Gallagher wrote:
On 02/21/2014 11:29 AM, Simo Sorce wrote:
> On Thu, 2014-02-13 at 09:39 -0700, Kevin Fenzi wrote:
>> On Tue, 11 Feb 2014 18:29:37 +0000 Colin Walters
>> <walters(a)verbum.org> wrote:
>>
>>> Hi,
>>>
>>> What about discussing implementation after we choose a few
>>> roles?
>>>
>>> I think FreeIPA has been mentioned before - it would make a
>>> good first featured role, and I think it's also probably
>>> something to which one would reasonably dedicate a machine.
>>> (As opposed to e.g. putting it in a Docker)
>>
>> Well, we already decided our first role would be a freeipa
>> server. ;)
>>
>> I don't know how far we want to go on the first cut, but it might
>> be worth picking 1 or 2 more so we can also look at how roles
>> interact.
>
> What would be the next thing you instal once you have a netowrk
> and identity infrastructure?
>
> To be honest I'd refrain from "fileserver" now because I know how
> complicated storage becomes very quickly, so before we tackle such
> a role I think we need to work on a few easier ones that do not
> have the huge variability a "file server" may have.
>
So let's take a look at what people use Fedora (or RHEL) servers for
today.
The server platform is there so that people can deploy their own apps
on it. So for us to have value, our Roles should be providing tools
that make these apps simpler. So let's look at commonalities.
Most apps these days need to talk to a database of some sort (be it
traditional like MariaDB or PostgreSQL or the new stuff like MongoDB).
A lot of web applications these days will also use a centralized
memcached host to accelerate their responses.
Naturally, there's also the plain old 'webserver', but that's probably
in a similar level of complexity to "fileserver", so we may want to
hold off on that for now.
I'd suggest that the second Role we might want to offer would be a
standard Database role. We can pick either MariaDB or PostgreSQL (by
majority vote) and run with it. (That's not to say we couldn't offer
both eventually, but let's not split our attention initially).
This should be relatively straightforward since setting up the
databases really requires only configuring authentication and starting
up the service. We can do a little integration work to ensure that the
auth works reasonably well with our identity management system and
then work on a configuration API to simplify creating new databases
and permissions on them. Probably not an insurmountable amount of work.
Thoughts?
I like this idea. And right now I have mariadb running on a bunch of
separate systems, which kinda sucks, so I could certainly dogfood it.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
12:08 p.m.
On Fri, 21 Feb 2014 10:00:36 -0800
Adam Williamson <awilliam(a)redhat.com> wrote:
I like this idea. And right now I have mariadb running on a bunch of
separate systems, which kinda sucks, so I could certainly dogfood it.
Ditto here, although I prefer postgres. ;)
There's a lot of value add we could put on top of a database role that
lots of people have to do themselves. Things like regular dumps of data
for backups, etc.
kevin
12:27 p.m.
On Fri, 2014-02-21 at 11:08 -0700, Kevin Fenzi wrote:
On Fri, 21 Feb 2014 10:00:36 -0800
Adam Williamson <awilliam(a)redhat.com> wrote:
> I like this idea. And right now I have mariadb running on a bunch of
> separate systems, which kinda sucks, so I could certainly dogfood it.
Ditto here, although I prefer postgres. ;)
+1 for postgres :-)
(and now you probably guess which is the db I hate due to data
losses :-)
There's a lot of value add we could put on top of a database role
that
lots of people have to do themselves. Things like regular dumps of data
for backups, etc.
Yes, I think it is fundamental for a role like this to enable backups,
if not by default as a very simple role configuration operation.
I do this for my postgres sql database and every time I need to redo it
I have forgotten everything and have to go back to learn how to
correctly run the dump command to get all the needed data.
ATM I also play with storing the dumps in git, for easy rollback :-P
Simo.
--
Simo Sorce * Red Hat, Inc * New York
3:40 a.m.
On Mon, 27 Jan 2014 11:33:30 -0700
Kevin Fenzi <kevin(a)scrye.com> wrote:
* We add comps groups for each featured server role we support. So,
for example: 'server-freeipa'. This comps group has all the packages
listed we want to install to setup the role as we determine is best.
This may include packages that are not direct dependencies on the
server pacakges.
Would say "fileserver"
include cifs,nfs,openssh to give resulting choice of client connection.
Or would it be one over the others.
___
Regards
Frank
frankly3d.com
2:57 p.m.
(I apologize for getting to this so late.)
2014-01-27 19:33 GMT+01:00 Kevin Fenzi <kevin(a)scrye.com>:
* The roles use cockpit to manage and configure the server role post
install.
Cockpit should be *a* mechanism, but not the only mechanism:
fully-automated, kickstart-driven installs need to be possible. One thing
that would work is an executable (fedora-role-deploy?) that accepts a
possibly empty kickstart-like service-specific "deployment configuration";
kickstart can invoke this directly, perhaps with a here-doc, and cockpit
can generate the configuration from a GUI and call that executable. To get
even more fancy, fedora-role-deploy could interactively ask if something is
missing in the deployment configuration, but that's perhaps inviting too
much complexity.
* We create a new fedora-server-$role package for each role. This role
package contains the 'glue' we add to turn this into a
fedora server
role from just a install with the packages and local config. We will
need to come up with guidelines and what these packages will contain,
but:
7. We may have a fedora-server-base or fedora-server-common package
to contain common/base stuff all roles need/user and make the role
packages require it. We would need to be very careful about updates
of this however.
Ability to cleanly upgrade is a significant concern here. I think the
worst case is a major update to something like a database, involving a
format change, where you need to keep the old version running to do a data
dump, and have the new version installed at the same time to receive the
data dump. That would require:
- The old and new version of the underlying software (not necessarily
the role) need to be parallel-installable. (This does not mean that
*every* two versions of the underlying software need to be
parallel-installable.)
- A role package that defines the new version needs to be installable
while the old version of the underlying software is still running. So, no
Conflicts:, at least not always. To extend this, the role system might
simultaneously want to still be able to handle the old version of the role,
so that e.g. monitoring or backups still work while the dump is in progress.
Another desirable feature would be to have a non-destructive upgrade
opportunity: install a newer version of the role package, ask it to do an
upgrade, and have the role package refuse an upgrade because it would lead
to data loss (or require non-trivial downtime or administrator's manual
involvement).
So, I'm leaning towards a design where fedora-server-$role is not a package
for a *specific version* of a role, but for *all past versions*. Therefore:
- The user can always (yum update) a role package, non-interactively,
without actually deploying an updated version of a role.
- "Minor" updates (no functionality loss, no human involvement) could
happen automatically from a %post or %posttrans script.
- (fedora-role-deploy --upgrade) or something similar could be used in
an interactive context (CLI / Cockpit) to actually perform a major update,
with the above-mentioned ability of a role to perform a feasibility check,
to ask for more information, or to confirm that the administrator can
accept a downtime needed for non-trivial migration (asking interactively or
using an updated deployment configuration file provided as input).
Downsides:
- We would need another mechanism to notify the user that the
interactive package update is insufficient and (fedora-role-deploy
--upgrade) needs to be run. (Or would we limit the major upgrades only to
new Fedora Server releases?)
- fedora-server-$role could not depend on versions of the underlying
packages that it supports; it would need to install them from
fedora-role-deploy in the worst case. (In the common case, the user would
ask for a $role comps group, and get both fedora-server-$role and the
appropriate versions of the underlying packages.)
What do you think?
Mirek
4:01 p.m.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 03/03/2014 03:57 PM, Miloslav Trmač wrote:
(I apologize for getting to this so late.)
2014-01-27 19:33 GMT+01:00 Kevin Fenzi <kevin(a)scrye.com
<mailto:kevin@scrye.com>>:
* The roles use cockpit to manage and configure the server role
post install.
Cockpit should be /a/ mechanism, but not the only mechanism:
Well, we've also been talking about having Cockpit provide the client
side deployment API, so we should probably consider different
terminology. Certainly Cockpit-the-webui must not be the only approach.
fully-automated, kickstart-driven installs need to be possible.
One thing that would work is an executable (fedora-role-deploy?)
that accepts a possibly empty kickstart-like service-specific
"deployment configuration"; kickstart can invoke this directly,
perhaps with a here-doc, and cockpit can generate the configuration
from a GUI and call that executable. To get even more fancy,
fedora-role-deploy could interactively ask if something is missing
in the deployment configuration, but that's perhaps inviting too
much complexity.
I'm not sure exactly where the GUI is coming into play here, but the
way I envisioned things was that we'd have a command-line tool that
could talk the D-BUS API that we could invoke in the kickstart %post.
This of course assumes that such a D-BUS API listener can be running
in %post; I'm unclear on that.
We would also need to make it clear that the fedora-server-$role would
have to be in the %packages section, since installing packages in
%post is a Bad Idea.
* We create a new fedora-server-$role package for each role. This
role package contains the 'glue' we add to turn this into a fedora
server role from just a install with the packages and local config.
We will need to come up with guidelines and what these packages
will contain, but:
7. We may have a fedora-server-base or fedora-server-common
package to contain common/base stuff all roles need/user and make
the role packages require it. We would need to be very careful
about updates of this however.
Ability to cleanly upgrade is a significant concern here. I think
the worst case is a major update to something like a database,
involving a format change, where you need to keep the old version
running to do a data dump, and have the new version installed at
the same time to receive the data dump. That would require:
* The old and new version of the underlying software (not
necessarily the role) need to be parallel-installable. (This does
not mean that /every/ two versions of the underlying software need
to be parallel-installable.) * A role package that defines the new
version needs to be installable while the old version of the
underlying software is still running. So, no Conflicts:, at least
not always. To extend this, the role system might simultaneously
want to still be able to handle the old version of the role, so
that e.g. monitoring or backups still work while the dump is in
progress.
Major upgrades with format changes are definitely one of the larger
question marks left in this effort and certainly not something we can
punt on and hope doesn't come up.
You are probably right that we need to guarantee both
parallel-installability as well as the ability for them both to be
running at the same time (since I can certainly foresee things like a
major FreeIPA upgrade being managed by a replication service instead
of a dump-and-reload).
Unfortunately, at the moment I have nothing constructive to add to
this. I could say something hand-wavy about containers, but I don't
think that's going to be feasible for F21 at least. (And I'm not sure
if it's a solution or a hack).
Another desirable feature would be to have a non-destructive
upgrade opportunity: install a newer version of the role package,
ask it to do an upgrade, and have the role package refuse an
upgrade because it would lead to data loss (or require non-trivial
downtime or administrator's manual involvement).
Yeah, I think this is clearly something we want and need to design
for. Of course, when selecting Roles to focus on, we should also be
examining a project's history for how it handles dangerous upgrades
(particularly frequency).
So, I'm leaning towards a design where fedora-server-$role is
not
a package for a /specific version/ of a role, but for /all past
versions/. Therefore:
* The user can always (yum update) a role package,
non-interactively, without actually deploying an updated version of
a role.
I'm really not sure what the mechanics of this would look like. Could
you go into some more detail here?
* "Minor" updates (no functionality loss, no human
involvement)
could happen automatically from a %post or %posttrans script.
Please no.
I, for one, have been working hard these last weeks on eliminating
scripts from packages wherever possible. %post scripts make life very
difficult for things like virt-sysprep and the proposed Fedora Atomic
initiative. I'd like for us to be using systemd drop-directory
snippets instead, so we can do upgrades and the like on ExecStartPre
triggers instead.
* (fedora-role-deploy --upgrade) or something similar could be used
in an interactive context (CLI / Cockpit) to actually perform a
major update, with the above-mentioned ability of a role to perform
a feasibility check, to ask for more information, or to confirm
that the administrator can accept a downtime needed for
non-trivial migration (asking interactively or using an updated
deployment configuration file provided as input).
This would be nice-to-have, but until I see some more concrete
information about how one might handle this at the filesystem and
service level, I remain skeptical.
Downsides:
* We would need another mechanism to notify the user that the
interactive package update is insufficient and (fedora-role-deploy
--upgrade) needs to be run. (Or would we limit the major upgrades
only to new Fedora Server releases?)
I'm definitely +1 on limiting major upgrades to new Server releases
(until and unless we start deploying Roles in fully contained
environments like a VM or kernel namespacing).
* fedora-server-$role could not depend on versions of the
underlying packages that it supports; it would need to install them
from fedora-role-deploy in the worst case. (In the common case,
the user would ask for a $role comps group, and get both
fedora-server-$role and the appropriate versions of the underlying
packages.)
Mandating that packages are installed from fedora-role-deploy would
have serious negative repercussions on trying to do things in anaconda
%post.
What do you think? Mirek
I'd like to see some of the above details filled in. I'm hesitant to
make a judgement about this yet.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEARECAAYFAlMY8DQACgkQeiVVYja6o6MNdwCgqn8o24QCtL9fCqHDhKhjevQx
iVsAn1mTOD94TmoQeNPYWLiUp/P9bHSO
=gEGO
-----END PGP SIGNATURE-----
5:59 p.m.
On Thu, Mar 06, 2014 at 05:01:24PM -0500, Stephen Gallagher wrote:
I'm not sure exactly where the GUI is coming into play here, but
the
way I envisioned things was that we'd have a command-line tool that
could talk the D-BUS API that we could invoke in the kickstart %post.
This of course assumes that such a D-BUS API listener can be running
in %post; I'm unclear on that.
This won't work, I'm afraid. The %post runs chrooted into the anaconda
environment, not in the installed one.
--
Matthew Miller -- Fedora Project -- <mattdm(a)fedoraproject.org>
8:08 a.m.
2014-03-07 0:59 GMT+01:00 Matthew Miller <mattdm(a)fedoraproject.org>:
On Thu, Mar 06, 2014 at 05:01:24PM -0500, Stephen Gallagher wrote:
> I'm not sure exactly where the GUI is coming into play here, but the
> way I envisioned things was that we'd have a command-line tool that
> could talk the D-BUS API that we could invoke in the kickstart %post.
> This of course assumes that such a D-BUS API listener can be running
> in %post; I'm unclear on that.
This won't work, I'm afraid. The %post runs chrooted into the anaconda
environment, not in the installed one.
http://fedoraproject.org/wiki/Anaconda/Kickstart#Chapter_5._Post-installa...
the opposite. (There may be other reasons why running D-Bus from the
chroot may be a problem, I don't know.)
Mirek
8:12 a.m.
On 07.03.2014 00:59, Matthew Miller wrote:
On Thu, Mar 06, 2014 at 05:01:24PM -0500, Stephen Gallagher wrote:
> I'm not sure exactly where the GUI is coming into play here, but the
> way I envisioned things was that we'd have a command-line tool that
> could talk the D-BUS API that we could invoke in the kickstart %post.
> This of course assumes that such a D-BUS API listener can be running
> in %post; I'm unclear on that.
This won't work, I'm afraid. The %post runs chrooted into the anaconda
environment, not in the installed one.
I've done it with realmd, where we chroot the daemon and use p2p dbus
... and it works well. You do have to take it into consideration when
building the DBus service.
Cheers,
Stef
8:31 a.m.
2014-03-06 23:01 GMT+01:00 Stephen Gallagher <sgallagh(a)redhat.com>:
> So, I'm leaning towards a design where fedora-server-$role
is not
> a package for a /specific version/ of a role, but for /all past
> versions/. Therefore:
>
> * The user can always (yum update) a role package,
> non-interactively, without actually deploying an updated version of
> a role.
I'm really not sure what the mechanics of this would look like. Could
you go into some more detail here?
fedora-server-role-postgresql.rpm contains all of:
/usr/share/fedora-roles/postgresql/__init__.py # common metadata
/usr/share/fedora-roles/postgresql/1/role.py # implementation, major
version 1
/usr/share/fedora-roles/postgresql/2/role.py # implementation, major
version 2
/usr/share/fedora-roles/postgresql/3/role.py # implementation, major
version 3
Adding a major version would add a subdirectory / subclass, not remove any
of the previous ones. (Well, we would probably feel the pressure to remove
the very old ones so that we don't have to keep porting them to new Python
versions and the like.)
* "Minor" updates (no functionality loss, no human
involvement)
> could happen automatically from a %post or %posttrans script.
Please no.
I, for one, have been working hard these last weeks on eliminating
scripts from packages wherever possible. %post scripts make life very
difficult for things like virt-sysprep and the proposed Fedora Atomic
initiative.
So %post vs. non-%post and automated vs. maual is a completely separate
discussion. We seem to be in agreement about actually making minor updates
automatically, are we?
Nevertheless... I don't think "%post is bad" approach makes sense. We can
have stateless storage of software (as opposed to configuration or data),
but *the computer* is not stateless!
Moving the scripts to auto-started systemd units still means that there are
scripts affecting state, only they are now harder to find and harder to
manage.
Also, the restriction to stateless storage of software is limiting and
leads to optimizing things the wrong way: we save time and complexity
during software installation ("only place a file in this directory") and
pay for it on every application start ("read all the files, resole
conflicts, build an in-memory index for speed") or in maintenance overhead
("don't forget to run update-cache-file or your changes won't be
visible"). Having an API
"add-this-information-to-the-system-wide-database" invoked during
installation would allow more validation checking, and maintaining the
information in a file format optimized for run-time efficiency.
* (fedora-role-deploy --upgrade) or something similar could be used
> in an interactive context (CLI / Cockpit) to actually perform a
> major update, with the above-mentioned ability of a role to perform
> a feasibility check, to ask for more information, or to confirm
> that the administrator can accept a downtime needed for
> non-trivial migration (asking interactively or using an updated
> deployment configuration file provided as input).
This would be nice-to-have, but until I see some more concrete
information about how one might handle this at the filesystem and
service level, I remain skeptical.
I'm not sure what information you are looking for; below is what I was
thinking of in more detail, though perhaps this is all trivial.
From the POV of the shared infrastructure:
# role_name = "postgresql"
role = System.get_role(role_name)
installed_version = role.installed_major_version
latest_version = role.latest_major_version
while installed_version < latest_version:
next_version = installed_version + 1
role_major = role.major_version[next_version]
try:
role_major.upgrade(role, deployment_descriptor) # checking and
possible interaction happens here
except (UpgradeAborted, UpgradeFailed):
break
role.installed_major_version = next_version
installed_version = next_version
(i.e. we only need to support upgrades "one step at a time")
From the POV of the role implementation, the "upgrade"
method would be
driving all of this (assuming postgresql-$old_version and
postgresql-$new_version can be installed in parallel):
- (Optionally) verify consistency of the current installation
- (Optionally) verify features are supported in the new version
- (Optionally) ask the user for confirmation if migration has not been
pre-approved in the deployment descriptor
- Ensure that postgresql-$the_next_version is installed
- Tell postgresql-$old_version to start refusing write requests, or just
stop the postgresql-$old_version service altogether
- Migrate data; whether this is in-place or to a copy is up to the role
implementation.
- Stop and disable postgresql-$old_version, start and enable
postgresql-$new_version
* fedora-server-$role could not depend on versions of the
> underlying packages that it supports; it would need to install them
> from fedora-role-deploy in the worst case. (In the common case,
> the user would ask for a $role comps group, and get both
> fedora-server-$role and the appropriate versions of the underlying
> packages.)
>
Mandating that packages are installed from fedora-role-deploy would
have serious negative repercussions on trying to do things in anaconda
%post.
For the anaconda %post case, the user would be expected to list an
appropriate comps group in %packages.
Mirek
3696
days inactive
3735
days old
server@lists.fedoraproject.org
33 comments
13 participants
participants (13)
-
Adam Williamson -
Alek Paunov -
Colin Walters -
Frank Murphy -
Kevin Fenzi -
Matthew Miller -
Michael R. Davis -
Miloslav Trmač -
poma -
Shawn Starr -
Simo Sorce -
Stef Walter -
Stephen Gallagher