One of the things that QA is focusing on ATM is getting better automation support for Fedora. We're currently doing this in two ways (that will eventually combine):
- taskotron (replacing autoqa) - beaker (different type of system - used by Red Hat QE, who is interested in running some of their tests upstream in Fedora)
Before we start granting access to the systems to Fedora contributors, we want to a) make the process relatively easy and b) isolate the client machines so that if something goes wrong or a bad actor gains access to them, any potential problems can be limited.
To do this, I'd like to propose one change and ask for input on another.
The first change is with FAS groups and shell access. To the best of my knowledge, in order to ssh into the bastion hosts, a user needs to be a member of the sysadmin group. While this works well for sysadmin people, I'm not sure I see a benefit of sending all sysadmin email to folks who are just looking to debug beaker or taskotron clients. I'd like to propose the creation of a new FAS shell group for qa automation and separate groups for beaker users, beaker admins, taskotron users and taskotron admins (qa-automation, beaker, beaker-admin, taskotron, taskotron-admin).
The qa-automation group wouldn't need access to bastion01, just whichever bastion host has access to the automation clients (bastion-comm01 for now). The other groups would be kind of like the relationship between the various sysadmin FAS groups and the base sysadmin group (ie, sysadmin access for all members, sysadmin-dns, sysadmin-qa etc. for farther access).
Does this seem reasonable and do-able? If so, I'd like to get the FAS group stuff done in the relatively near future - not "drop everything and do it now" but ideally in the next couple of weeks if possible.
For network isolation, I don't pretend to be an expert on networking so I'll describe the functionality that we're looking for and what I think might work for a solution, but I'll defer to the expertise here on whether it's a good idea or not :)
The beaker and taskotron clients will need network access to several Fedora systems in order to work.
Taskotron Clients: - Taskotron buildmaster - bodhi, koji, repos, dist-git, task-git (part of taskotron, not yet created), resultsdb (also part of taskotron)
Beaker Clients: - Beaker server and lab controller (same system for now) - repos, maybe grabbing packages from koji/bodhi
I put together a quick diagram with the various network connections: http://tflink.fedorapeople.org/taskotron/client-network-connections.png
From a few previous conversations, I think that a private network for the clients could provide the isolation that we're looking for. As far as getting network access to the systems needed to function, I figured that the beaker server and taskotron master would have network interfaces on this private network and a gateway could be used to restrict outgoing traffic to only the resources required.
All of the clients would be hosted on the qa virthosts, which are currently in the same rack. I was thinking that it would be possible to use one of the network interfaces in these virthosts to create this private network (assuming that the network switch capacity is available) but I'm definitely open to other ideas.
Does this idea for network access and isolation seem reasonable and do-able? I figure that the network isolation/access part will require more discussion and time for implementation after a decision is reached. Our systems will work fine with the current network configuration but I wanted to get this part of the conversation started so that the implementation could happen before we get too far with automation development.
Thanks,
Tim
On Fri, 10 Jan 2014 11:09:16 -0700 Tim Flink tflink@redhat.com wrote:
One of the things that QA is focusing on ATM is getting better automation support for Fedora. We're currently doing this in two ways (that will eventually combine):
- taskotron (replacing autoqa)
- beaker (different type of system - used by Red Hat QE, who is interested in running some of their tests upstream in Fedora)
Before we start granting access to the systems to Fedora contributors, we want to a) make the process relatively easy and b) isolate the client machines so that if something goes wrong or a bad actor gains access to them, any potential problems can be limited.
Absolutely. :)
To do this, I'd like to propose one change and ask for input on another.
The first change is with FAS groups and shell access. To the best of my knowledge, in order to ssh into the bastion hosts, a user needs to be a member of the sysadmin group. While this works well for sysadmin people, I'm not sure I see a benefit of sending all sysadmin email to folks who are just looking to debug beaker or taskotron clients. I'd like to propose the creation of a new FAS shell group for qa automation and separate groups for beaker users, beaker admins, taskotron users and taskotron admins (qa-automation, beaker, beaker-admin, taskotron, taskotron-admin).
So, it's a bit more complex than that. ;)
All 'sysadmin-*' groups require you to be a member of 'sysadmin' first. This means if you are being added to 'sysadmin-qa' or something, you have to first get added to sysadmin. sysadmin is a tracking group, it provides no shell access to anything, but it does provide other things:
* nagios alerts * all commits from puppet/ansible/etc repos via email * Can push commits to puppet/ansible/etc repos (but need to be in some other group that provides them shell access to bastion and then lockbox01 where the repos are).
So, the idea is that anyone is a sysadmin subgroup gets all the info and commits and can see whats going on, etc.
The qa-automation group wouldn't need access to bastion01, just whichever bastion host has access to the automation clients (bastion-comm01 for now). The other groups would be kind of like the relationship between the various sysadmin FAS groups and the base sysadmin group (ie, sysadmin access for all members, sysadmin-dns, sysadmin-qa etc. for farther access).
Does this seem reasonable and do-able? If so, I'd like to get the FAS group stuff done in the relatively near future - not "drop everything and do it now" but ideally in the next couple of weeks if possible.
I have no objection. When we setup the sysadmin-qa group we made it like the rest of the sysadmin groups, but mostly that was due to me not being sure there wasn't something else going on with sysadmin group. :)
I'm not sure you need admin and users groups seperate? Could you just have one group and make some people admins in the group and key off that?
I think yeah, bastion-comm01 should mostly work fine, except if people in those groups need to commit to ansible/puppet/etc repos. Would they? Or would that be something we keep with sysadmin-qa?
For network isolation, I don't pretend to be an expert on networking so I'll describe the functionality that we're looking for and what I think might work for a solution, but I'll defer to the expertise here on whether it's a good idea or not :)
The beaker and taskotron clients will need network access to several Fedora systems in order to work.
Taskotron Clients:
- Taskotron buildmaster
- bodhi, koji, repos, dist-git, task-git (part of taskotron, not yet created), resultsdb (also part of taskotron)
Beaker Clients:
- Beaker server and lab controller (same system for now)
- repos, maybe grabbing packages from koji/bodhi
I put together a quick diagram with the various network connections: http://tflink.fedorapeople.org/taskotron/client-network-connections.png
From a few previous conversations, I think that a private network for the clients could provide the isolation that we're looking for. As far as getting network access to the systems needed to function, I figured that the beaker server and taskotron master would have network interfaces on this private network and a gateway could be used to restrict outgoing traffic to only the resources required.
All of the clients would be hosted on the qa virthosts, which are currently in the same rack. I was thinking that it would be possible to use one of the network interfaces in these virthosts to create this private network (assuming that the network switch capacity is available) but I'm definitely open to other ideas.
Does this idea for network access and isolation seem reasonable and do-able? I figure that the network isolation/access part will require more discussion and time for implementation after a decision is reached. Our systems will work fine with the current network configuration but I wanted to get this part of the conversation started so that the implementation could happen before we get too far with automation development.
Let me ponder on this part... part of the problem is that we do not control switches or routers. We have to ask RHIT about those... so we can't just do things and it's always better when we do something that requires them to not have to do anything. ;)
kevin
On Fri, 10 Jan 2014 11:29:40 -0700 Kevin Fenzi kevin@scrye.com wrote:
On Fri, 10 Jan 2014 11:09:16 -0700 Tim Flink tflink@redhat.com wrote:
One of the things that QA is focusing on ATM is getting better automation support for Fedora. We're currently doing this in two ways (that will eventually combine):
- taskotron (replacing autoqa)
- beaker (different type of system - used by Red Hat QE, who is interested in running some of their tests upstream in Fedora)
Before we start granting access to the systems to Fedora contributors, we want to a) make the process relatively easy and b) isolate the client machines so that if something goes wrong or a bad actor gains access to them, any potential problems can be limited.
Absolutely. :)
To do this, I'd like to propose one change and ask for input on another.
The first change is with FAS groups and shell access. To the best of my knowledge, in order to ssh into the bastion hosts, a user needs to be a member of the sysadmin group. While this works well for sysadmin people, I'm not sure I see a benefit of sending all sysadmin email to folks who are just looking to debug beaker or taskotron clients. I'd like to propose the creation of a new FAS shell group for qa automation and separate groups for beaker users, beaker admins, taskotron users and taskotron admins (qa-automation, beaker, beaker-admin, taskotron, taskotron-admin).
So, it's a bit more complex than that. ;)
All 'sysadmin-*' groups require you to be a member of 'sysadmin' first. This means if you are being added to 'sysadmin-qa' or something, you have to first get added to sysadmin. sysadmin is a tracking group, it provides no shell access to anything, but it does provide other things:
- nagios alerts
- all commits from puppet/ansible/etc repos via email
- Can push commits to puppet/ansible/etc repos (but need to be in some other group that provides them shell access to bastion and then lockbox01 where the repos are).
So, the idea is that anyone is a sysadmin subgroup gets all the info and commits and can see whats going on, etc.
Ha, I'm pretty sure I've been corrected on this misunderstanding before and forgot. Thanks for the reminder :)
The qa-automation group wouldn't need access to bastion01, just whichever bastion host has access to the automation clients (bastion-comm01 for now). The other groups would be kind of like the relationship between the various sysadmin FAS groups and the base sysadmin group (ie, sysadmin access for all members, sysadmin-dns, sysadmin-qa etc. for farther access).
Does this seem reasonable and do-able? If so, I'd like to get the FAS group stuff done in the relatively near future - not "drop everything and do it now" but ideally in the next couple of weeks if possible.
I have no objection. When we setup the sysadmin-qa group we made it like the rest of the sysadmin groups, but mostly that was due to me not being sure there wasn't something else going on with sysadmin group. :)
I'm not sure you need admin and users groups seperate? Could you just have one group and make some people admins in the group and key off that?
For the moment, it doesn't matter much beyond shell access because we don't have FAS integration for either taskotron or beaker. There are admin tasks for beaker that and there will be admin tasks for taskotron. I'd prefer not to have more interfaces for group/permission management than we need to have but as long as we can manage shell access and base group membership to the two systems, we'll make it work.
I think yeah, bastion-comm01 should mostly work fine, except if people in those groups need to commit to ansible/puppet/etc repos. Would they? Or would that be something we keep with sysadmin-qa?
The taskotron user and admin groups would be separate from any sysadmin groups - I'd like to keep sysadmin-qa separate. I don't want to be giving shell access to all our virthosts, write access to the infra playbooks etc. to every user of beaker and taskotron who needs shell access to the clients.
For network isolation, I don't pretend to be an expert on networking so I'll describe the functionality that we're looking for and what I think might work for a solution, but I'll defer to the expertise here on whether it's a good idea or not :)
The beaker and taskotron clients will need network access to several Fedora systems in order to work.
Taskotron Clients:
- Taskotron buildmaster
- bodhi, koji, repos, dist-git, task-git (part of taskotron, not
yet created), resultsdb (also part of taskotron)
Beaker Clients:
- Beaker server and lab controller (same system for now)
- repos, maybe grabbing packages from koji/bodhi
I put together a quick diagram with the various network connections: http://tflink.fedorapeople.org/taskotron/client-network-connections.png
From a few previous conversations, I think that a private network for the clients could provide the isolation that we're looking for. As far as getting network access to the systems needed to function, I figured that the beaker server and taskotron master would have network interfaces on this private network and a gateway could be used to restrict outgoing traffic to only the resources required.
All of the clients would be hosted on the qa virthosts, which are currently in the same rack. I was thinking that it would be possible to use one of the network interfaces in these virthosts to create this private network (assuming that the network switch capacity is available) but I'm definitely open to other ideas.
Does this idea for network access and isolation seem reasonable and do-able? I figure that the network isolation/access part will require more discussion and time for implementation after a decision is reached. Our systems will work fine with the current network configuration but I wanted to get this part of the conversation started so that the implementation could happen before we get too far with automation development.
Let me ponder on this part... part of the problem is that we do not control switches or routers. We have to ask RHIT about those... so we can't just do things and it's always better when we do something that requires them to not have to do anything. ;)
No worries, I didn't think that any networking changes would be immediate. I don't have my heart set on that exact solution, either. It just made the most sense to me. If there are easier alternatives that accomplish the same goals of isolation, I'm game.
Tim
On Fri, 10 Jan 2014 11:09:16 -0700 Tim Flink tflink@redhat.com wrote:
For network isolation, I don't pretend to be an expert on networking so I'll describe the functionality that we're looking for and what I think might work for a solution, but I'll defer to the expertise here on whether it's a good idea or not :)
The beaker and taskotron clients will need network access to several Fedora systems in order to work.
Taskotron Clients:
- Taskotron buildmaster
- bodhi, koji, repos, dist-git, task-git (part of taskotron, not yet created), resultsdb (also part of taskotron)
Beaker Clients:
- Beaker server and lab controller (same system for now)
- repos, maybe grabbing packages from koji/bodhi
ok, and to be clear the koji/bodhi/dist-git is all public stuff right? (ie, it could get it via public ip ok?)
I put together a quick diagram with the various network connections: http://tflink.fedorapeople.org/taskotron/client-network-connections.png
Cool. All those arrows are bidirectional? Are all the ones outside the box http/https?
From a few previous conversations, I think that a private network for the clients could provide the isolation that we're looking for. As far as getting network access to the systems needed to function, I figured that the beaker server and taskotron master would have network interfaces on this private network and a gateway could be used to restrict outgoing traffic to only the resources required.
So, in some senses the 'qa' network is this. It's restricted from talking to other internal stuff with some exceptions.
Sadly over time, we have grown the number of things in that network and of course all the stuff in that network can talk to each other (barring local firewalls).
All of the clients would be hosted on the qa virthosts, which are currently in the same rack. I was thinking that it would be possible to use one of the network interfaces in these virthosts to create this private network (assuming that the network switch capacity is available) but I'm definitely open to other ideas.
Could we just do it with a private libvirt network on the qa virthosts? ie, pick 172.31.17.0 and put them all in that and setup a bastion host as their gateway that does NAT for them out to the stuff they need. Or would NAT not work for this? They would still physically be on the qa network tho, so I guess we could try and request a real seperate one from RHIT.
Does this idea for network access and isolation seem reasonable and do-able? I figure that the network isolation/access part will require more discussion and time for implementation after a decision is reached. Our systems will work fine with the current network configuration but I wanted to get this part of the conversation started so that the implementation could happen before we get too far with automation development.
Yeah, I think we can make something work here.
There was also talk about redoing a lot of our network setup a while back, but not sure where that went. The thought was to completely seperate Fedora from anything else (which would be great), but would require rework on a bunch of things. Once it's done however, we could not have to care as much about adding new private nets, etc.
kevin
On Tue, 14 Jan 2014 16:09:05 -0700 Kevin Fenzi kevin@scrye.com wrote:
On Fri, 10 Jan 2014 11:09:16 -0700 Tim Flink tflink@redhat.com wrote:
For network isolation, I don't pretend to be an expert on networking so I'll describe the functionality that we're looking for and what I think might work for a solution, but I'll defer to the expertise here on whether it's a good idea or not :)
The beaker and taskotron clients will need network access to several Fedora systems in order to work.
Taskotron Clients:
- Taskotron buildmaster
- bodhi, koji, repos, dist-git, task-git (part of taskotron, not
yet created), resultsdb (also part of taskotron)
Beaker Clients:
- Beaker server and lab controller (same system for now)
- repos, maybe grabbing packages from koji/bodhi
ok, and to be clear the koji/bodhi/dist-git is all public stuff right? (ie, it could get it via public ip ok?)
Yeah, it's all access to the public apis and URLs.
I put together a quick diagram with the various network connections: http://tflink.fedorapeople.org/taskotron/client-network-connections.png
Cool. All those arrows are bidirectional?
Anything outside the box is not - the clients need to be able to access those resources but those resources do not need to be able to access the clients.
Are all the ones outside the box http/https?
Yes. The only stuff that's not http/https is the connections from bastion, the beaker server and the taskotron master(s).
The one thing that isn't part of that diagram is some sort of lockbox-ish host for managing and monitoring the clients. I'm still not sure how we want to configure all that (we had talked about putting everything but the clients in the infra playbook on lockbox01 and managing the clients with a different playbook on a different host but none of that has been done), though but I suspect that is one of the _simpler_ parts of the setup.
From a few previous conversations, I think that a private network for the clients could provide the isolation that we're looking for. As far as getting network access to the systems needed to function, I figured that the beaker server and taskotron master would have network interfaces on this private network and a gateway could be used to restrict outgoing traffic to only the resources required.
So, in some senses the 'qa' network is this. It's restricted from talking to other internal stuff with some exceptions.
Sadly over time, we have grown the number of things in that network and of course all the stuff in that network can talk to each other (barring local firewalls).
Isn't that what usually happens to stuff like the qa network over time? :)
All of the clients would be hosted on the qa virthosts, which are currently in the same rack. I was thinking that it would be possible to use one of the network interfaces in these virthosts to create this private network (assuming that the network switch capacity is available) but I'm definitely open to other ideas.
Could we just do it with a private libvirt network on the qa virthosts? ie, pick 172.31.17.0 and put them all in that and setup a bastion host as their gateway that does NAT for them out to the stuff they need. Or would NAT not work for this? They would still physically be on the qa network tho, so I guess we could try and request a real seperate one from RHIT.
I'm not sure I understand this completely. Would this allow the various "special" hosts (beaker server and bastion host in particular) to connect to the clients without adding the step of "connect to virthostX" before being able to ssh into the clients?
NAT should work for everything other than communication with the beaker server and bastion hosts.
Another option might be to use ebtables. I haven't investigated this fully yet but it sounds like it would allow the use of qa network IPs but restrict the traffic running through the bridged network on the virthosts - effectively creating a restricted network without needing any physical changes. Of course, that would only work with VM clients but that's all we're planning for at the moment.
Does this idea for network access and isolation seem reasonable and do-able? I figure that the network isolation/access part will require more discussion and time for implementation after a decision is reached. Our systems will work fine with the current network configuration but I wanted to get this part of the conversation started so that the implementation could happen before we get too far with automation development.
Yeah, I think we can make something work here.
There was also talk about redoing a lot of our network setup a while back, but not sure where that went. The thought was to completely seperate Fedora from anything else (which would be great), but would require rework on a bunch of things. Once it's done however, we could not have to care as much about adding new private nets, etc.
That sounds like it would end up in a good place for isolating the clients but it also sounds like a lot of work. On the other hand, this "lull" between actively working on releases might be a "less bad" time to do major changes like that but I'll refrain from commenting more on it since I'd likely not be the one doing all the work.
Tim
On Wed, 15 Jan 2014 11:11:51 -0700 Tim Flink tflink@redhat.com wrote:
The one thing that isn't part of that diagram is some sort of lockbox-ish host for managing and monitoring the clients. I'm still not sure how we want to configure all that (we had talked about putting everything but the clients in the infra playbook on lockbox01 and managing the clients with a different playbook on a different host but none of that has been done), though but I suspect that is one of the _simpler_ parts of the setup.
Could be yeah. I am planning on making a lockbox01.qa after our conversation the other day... ...snip....
Could we just do it with a private libvirt network on the qa virthosts? ie, pick 172.31.17.0 and put them all in that and setup a bastion host as their gateway that does NAT for them out to the stuff they need. Or would NAT not work for this? They would still physically be on the qa network tho, so I guess we could try and request a real seperate one from RHIT.
I'm not sure I understand this completely. Would this allow the various "special" hosts (beaker server and bastion host in particular) to connect to the clients without adding the step of "connect to virthostX" before being able to ssh into the clients?
Sure, they would just use the bridge on the virthosts to send their private network stuff around, but it would be traveling on the same physical network as the 10.5.124.x qa network.
NAT should work for everything other than communication with the beaker server and bastion hosts.
Right.
Another option might be to use ebtables. I haven't investigated this fully yet but it sounds like it would allow the use of qa network IPs but restrict the traffic running through the bridged network on the virthosts - effectively creating a restricted network without needing any physical changes. Of course, that would only work with VM clients but that's all we're planning for at the moment.
Yeah, that was essentially what I was suggesting above actually. ;)
There was also talk about redoing a lot of our network setup a while back, but not sure where that went. The thought was to completely seperate Fedora from anything else (which would be great), but would require rework on a bunch of things. Once it's done however, we could not have to care as much about adding new private nets, etc.
That sounds like it would end up in a good place for isolating the clients but it also sounds like a lot of work. On the other hand, this "lull" between actively working on releases might be a "less bad" time to do major changes like that but I'll refrain from commenting more on it since I'd likely not be the one doing all the work.
Yeah. Lots of other things in the mix and it's also something that would have to weigh on RHIT networking folks more, so we will see.
kevin
On Sat, 18 Jan 2014 13:26:30 -0700 Kevin Fenzi kevin@scrye.com wrote:
On Wed, 15 Jan 2014 11:11:51 -0700 Tim Flink tflink@redhat.com wrote:
The one thing that isn't part of that diagram is some sort of lockbox-ish host for managing and monitoring the clients. I'm still not sure how we want to configure all that (we had talked about putting everything but the clients in the infra playbook on lockbox01 and managing the clients with a different playbook on a different host but none of that has been done), though but I suspect that is one of the _simpler_ parts of the setup.
Could be yeah. I am planning on making a lockbox01.qa after our conversation the other day...
Yeah, from that conversation, it seems like the lockbox01.qa is the best choice for now.
...snip....
Could we just do it with a private libvirt network on the qa virthosts? ie, pick 172.31.17.0 and put them all in that and setup a bastion host as their gateway that does NAT for them out to the stuff they need. Or would NAT not work for this? They would still physically be on the qa network tho, so I guess we could try and request a real seperate one from RHIT.
I'm not sure I understand this completely. Would this allow the various "special" hosts (beaker server and bastion host in particular) to connect to the clients without adding the step of "connect to virthostX" before being able to ssh into the clients?
Sure, they would just use the bridge on the virthosts to send their private network stuff around, but it would be traveling on the same physical network as the 10.5.124.x qa network.
As noted below, it sounds like I have some reading to do, but that sounds as if it would work.
NAT should work for everything other than communication with the beaker server and bastion hosts.
Right.
Another option might be to use ebtables. I haven't investigated this fully yet but it sounds like it would allow the use of qa network IPs but restrict the traffic running through the bridged network on the virthosts - effectively creating a restricted network without needing any physical changes. Of course, that would only work with VM clients but that's all we're planning for at the moment.
Yeah, that was essentially what I was suggesting above actually. ;)
Ah, at least we have similar solutions in mind :) I guess I'll have to do some more reading on libvirt networks. When I looked at the docs, it sounded like those were designed only for VMs on the same virthost - not communication between VMs on different virthosts.
There was also talk about redoing a lot of our network setup a while back, but not sure where that went. The thought was to completely seperate Fedora from anything else (which would be great), but would require rework on a bunch of things. Once it's done however, we could not have to care as much about adding new private nets, etc.
That sounds like it would end up in a good place for isolating the clients but it also sounds like a lot of work. On the other hand, this "lull" between actively working on releases might be a "less bad" time to do major changes like that but I'll refrain from commenting more on it since I'd likely not be the one doing all the work.
Yeah. Lots of other things in the mix and it's also something that would have to weigh on RHIT networking folks more, so we will see.
OK, we'll plan on finding something that works without any physical networking changes.
Tim
infrastructure@lists.fedoraproject.org
-
Kevin Fenzi -
Tim Flink