12:27 p.m.
We've run into an issue with travis where adding new make files causes
the tests to fail with an "write error" when running distclean.
For example: https://travis-ci.org/cockpit-project/cockpit/jobs/76408030.
We haven't been able to reproduce these failures anywhere else. In
addition their new container based infrastructure that they are moving
towards doesn't support sudo which we need for our reauthorize tests. So
we are looking at replacing travis with a different CI service.
I looked at semaphoreci and circleci. I was eventually able to get our
test suite to pass on both systems. Both seems to have most of the same
github integration / badges but Circleci is based on Ubuntu 12 and we'd
really like to be able to use something newer.
You can see my cockpit fork on semaphoreci here:
https://semaphoreci.com/petervo/cockpit
PROS:
- It works and uses a newer OS (Ubuntu 14)
- Allows you to ssh into instances to run / debug tests.
CONS:
- There is only one configuration per project that is configured via
their website. So you can't change the commands on a per branch / PR
basis. Additionally the webui has some bugs and usability issues that
end up requiring a few tries before it gets the commands in the right
order and in the right threads.
- We have to run on their container based beta platform as their normal
platform doesn't fully support inotify and causes tests to fail.
- Doesn't do IRC notifications out of the box. Though we could probably
implement something using webhooks if we wanted to.
What does everyone think about switching? Worth doing or should we keep
looking at other options?
1:54 a.m.
Peter <petervo(a)redhat.com> writes:
PROS:
- It works and uses a newer OS (Ubuntu 14)
- Allows you to ssh into instances to run / debug tests.
This is very, very nice.
CONS:
- There is only one configuration per project that is configured via
their website. So you can't change the commands on a per branch / PR
basis.
We could have a script in our sources that does all the work and
configure semaphoreci to just run that script. Would that work?
- We have to run on their container based beta platform as their
normal
platform doesn't fully support inotify and causes tests to fail.
This is weird. How do they manage to not fully support inotify? Is it
a weird/broken filesystem? Some absurdly low limits somewhere?
- Doesn't do IRC notifications out of the box. Though we could
probably
implement something using webhooks if we wanted to.
Personally, I didn't find the notifications very useful.
What does everyone think about switching? Worth doing or should we
keep looking at other options?
Right now the options are containers in either Semaphore or Travis,
right? (Which means we need to adjust the tests that need sudo, right?)
And switching to containers in Travis might just make the weird "write
error" bug disappear, I guess.
But since you have done the work, and Semaphore has the "ssh" killer
feature, I vote to switch to Semaphore.
2:03 a.m.
On 24.08.2015 08:54, Marius Vollmer wrote:
Peter <petervo(a)redhat.com> writes:
> PROS:
>
> - It works and uses a newer OS (Ubuntu 14)
> - Allows you to ssh into instances to run / debug tests.
This is very, very nice.
> CONS:
Obviously another big con is that this stuff is not Open Source. These
are proprietary services. But so is GitHub. But until we have more
resources, and/or an open source hosted CI service ... we'll probably
have to ignore this con.
> - There is only one configuration per project that is configured
via
> their website. So you can't change the commands on a per branch / PR
> basis.
We could have a script in our sources that does all the work and
configure semaphoreci to just run that script. Would that work?
> - We have to run on their container based beta platform as their normal
> platform doesn't fully support inotify and causes tests to fail.
This is weird. How do they manage to not fully support inotify? Is it
a weird/broken filesystem? Some absurdly low limits somewhere?
NFS.
> - Doesn't do IRC notifications out of the box. Though we
could probably
> implement something using webhooks if we wanted to.
Personally, I didn't find the notifications very useful.
> What does everyone think about switching? Worth doing or should we
> keep looking at other options?
Right now the options are containers in either Semaphore or Travis,
right? (Which means we need to adjust the tests that need sudo, right?)
sudo can run in a container. It's just the Travis containers that don't
allow this. Presumably because it's easier to break out of a container
if you have root.
And switching to containers in Travis might just make the weird
"write
error" bug disappear, I guess.
But more importantly the Travis containers have an ancient operating
system (from 2012) and can't be updated (due to no root access).
But since you have done the work, and Semaphore has the
"ssh" killer
feature, I vote to switch to Semaphore.
Me too.
Stef
3:55 a.m.
> This is weird. How do they manage to not fully support inotify?
Is it
> a weird/broken filesystem? Some absurdly low limits somewhere?
NFS.
Oh, I see.
sudo can run in a container. It's just the Travis containers that
don't
allow this. [...]
Ok, so Semaphore does allow it and we don't need to change the tests?
That is another big pro.
> And switching to containers in Travis might just make the weird
"write
> error" bug disappear, I guess.
But more importantly the Travis containers have an ancient operating
system (from 2012) and can't be updated (due to no root access).
Yes, the pros are very strong.
7:45 a.m.
On 24.08.2015 09:03, Stef Walter wrote:
On 24.08.2015 08:54, Marius Vollmer wrote:
> Peter <petervo(a)redhat.com> writes:
>
>> PROS:
>>
>> - It works and uses a newer OS (Ubuntu 14)
>> - Allows you to ssh into instances to run / debug tests.
>
> This is very, very nice.
>
>> CONS:
Obviously another big con is that this stuff is not Open Source. These
are proprietary services. But so is GitHub. But until we have more
resources, and/or an open source hosted CI service ... we'll probably
have to ignore this con.
I was wrong about Travis. Travis is Open Source. Yay. Good for them.
That changes how much effort I feel we should dedicate to helping make
it work, filing bugs, etc.
A little bird also whispered in my ear that Marius has found a
work-around for the Travis bug that has been bothering us.
Stef
8:15 a.m.
On Mon, Aug 24, 2015 at 8:45 AM, Stef Walter <stefw(a)redhat.com> wrote:
On 24.08.2015 09:03, Stef Walter wrote:
> On 24.08.2015 08:54, Marius Vollmer wrote:
>> Peter <petervo(a)redhat.com> writes:
>>
>>> PROS:
>>>
>>> - It works and uses a newer OS (Ubuntu 14)
>>> - Allows you to ssh into instances to run / debug tests.
>>
>> This is very, very nice.
>>
>>> CONS:
>
> Obviously another big con is that this stuff is not Open Source. These
> are proprietary services. But so is GitHub. But until we have more
> resources, and/or an open source hosted CI service ... we'll probably
> have to ignore this con.
I was wrong about Travis. Travis is Open Source. Yay. Good for them.
That changes how much effort I feel we should dedicate to helping make
it work, filing bugs, etc.
A little bird also whispered in my ear that Marius has found a
work-around for the Travis bug that has been bothering us.
Not sure if this would help, but the AOS team has a Jenkins server where we
could do some more sophisticated CI testing:
https://ci.openshift.redhat.com/jenkins/
I am one of the maintainers so I'd be happy to help set things up if this
was of interest.
Stef
_______________________________________________
cockpit-devel mailing list
cockpit-devel(a)lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/cockpit-devel
8:27 a.m.
On 24.08.2015 15:15, Harrison Ripps wrote:
On Mon, Aug 24, 2015 at 8:45 AM, Stef Walter <stefw(a)redhat.com
<mailto:stefw@redhat.com>> wrote:
On 24.08.2015 09 <tel:24.08.2015%2009>:03, Stef Walter wrote:
> On 24.08.2015 08 <tel:24.08.2015%2008>:54, Marius Vollmer wrote:
>> Peter <petervo(a)redhat.com <mailto:petervo@redhat.com>> writes:
>>
>>> PROS:
>>>
>>> - It works and uses a newer OS (Ubuntu 14)
>>> - Allows you to ssh into instances to run / debug tests.
>>
>> This is very, very nice.
>>
>>> CONS:
>
> Obviously another big con is that this stuff is not Open Source. These
> are proprietary services. But so is GitHub. But until we have more
> resources, and/or an open source hosted CI service ... we'll probably
> have to ignore this con.
I was wrong about Travis. Travis is Open Source. Yay. Good for them.
That changes how much effort I feel we should dedicate to helping make
it work, filing bugs, etc.
A little bird also whispered in my ear that Marius has found a
work-around for the Travis bug that has been bothering us.
Not sure if this would help, but the AOS team has a Jenkins server where
we could do some more sophisticated CI testing:
https://ci.openshift.redhat.com/jenkins/
I am one of the maintainers so I'd be happy to help set things up if
this was of interest.
Interesting. Have you migrated anything from Travis to that Jenkins
instance? What did the process end up looking like?
But this is actually more interesting for the workloads we don't run on
Travis ... our integration tests ... which launch tons of VMs within the
space of a few minutes. Is that the sort of thing your Jenkins instance
can run? ie: nested virtualization, and/or root access etc?
Cheers,
Stef
8:44 a.m.
On Mon, Aug 24, 2015 at 9:27 AM, Stef Walter <stefw(a)redhat.com> wrote:
On 24.08.2015 15:15, Harrison Ripps wrote:
>
> On Mon, Aug 24, 2015 at 8:45 AM, Stef Walter <stefw(a)redhat.com
> <mailto:stefw@redhat.com>> wrote:
>
> On 24.08.2015 09 <tel:24.08.2015%2009>:03, Stef Walter wrote:
> > On 24.08.2015 08 <tel:24.08.2015%2008>:54, Marius Vollmer wrote:
> >> Peter <petervo(a)redhat.com <mailto:petervo@redhat.com>>
writes:
> >>
> >>> PROS:
> >>>
> >>> - It works and uses a newer OS (Ubuntu 14)
> >>> - Allows you to ssh into instances to run / debug tests.
> >>
> >> This is very, very nice.
> >>
> >>> CONS:
> >
> > Obviously another big con is that this stuff is not Open Source.
These
> > are proprietary services. But so is GitHub. But until we have more
> > resources, and/or an open source hosted CI service ... we'll
probably
> > have to ignore this con.
>
> I was wrong about Travis. Travis is Open Source. Yay. Good for them.
> That changes how much effort I feel we should dedicate to helping
make
> it work, filing bugs, etc.
>
> A little bird also whispered in my ear that Marius has found a
> work-around for the Travis bug that has been bothering us.
>
>
> Not sure if this would help, but the AOS team has a Jenkins server where
> we could do some more sophisticated CI testing:
>
> https://ci.openshift.redhat.com/jenkins/
>
> I am one of the maintainers so I'd be happy to help set things up if
> this was of interest.
Interesting. Have you migrated anything from Travis to that Jenkins
instance? What did the process end up looking like?
I have not done a Travis to Jenkins migration. However, based on looking at
a reference Travis config[1], I'd say the porting is relatively painless.
The Jenkins host needs to have the right tools installed on it and then you
write a shell script for Jenkins to run.
[1]: https://github.com/openshift/origin/blob/master/.travis.yml
But this is actually more interesting for the workloads we don't
run on
Travis ... our integration tests ... which launch tons of VMs within the
space of a few minutes. Is that the sort of thing your Jenkins instance
can run? ie: nested virtualization, and/or root access etc?
The public AOS Jenkins system is running on an EC2 instance. With its
current workload, I don't think it has enough power to stand up a bunch of
locally hosted VMs for testing. However, if the tests could be modified to
spin up EC2 instances, or instances on the Red Hat public OpenStack
cluster, then the AOS Jenkins host could certainly handle the workload of
firing them up, running tests, and reporting on the test results.
Cheers,
Stef
_______________________________________________
cockpit-devel mailing list
cockpit-devel(a)lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/cockpit-devel
8:54 a.m.
On 24.08.2015 15:44, Harrison Ripps wrote:
On Mon, Aug 24, 2015 at 9:27 AM, Stef Walter <stefw(a)redhat.com
<mailto:stefw@redhat.com>> wrote:
On 24.08.2015 15 <tel:24.08.2015%2015>:15, Harrison Ripps wrote:
>
> On Mon, Aug 24, 2015 at 8:45 AM, Stef Walter <stefw(a)redhat.com
<mailto:stefw@redhat.com>
> <mailto:stefw@redhat.com <mailto:stefw@redhat.com>>> wrote:
>
> On 24.08.2015 09 <tel:24.08.2015%2009>
<tel:24.08.2015%2009>:03, Stef Walter wrote:
> > On 24.08.2015 08 <tel:24.08.2015%2008>
<tel:24.08.2015%2008>:54, Marius Vollmer wrote:
> >> Peter <petervo(a)redhat.com <mailto:petervo@redhat.com>
<mailto:petervo@redhat.com <mailto:petervo@redhat.com>>> writes:
> >>
> >>> PROS:
> >>>
> >>> - It works and uses a newer OS (Ubuntu 14)
> >>> - Allows you to ssh into instances to run / debug tests.
> >>
> >> This is very, very nice.
> >>
> >>> CONS:
> >
> > Obviously another big con is that this stuff is not Open Source. These
> > are proprietary services. But so is GitHub. But until we have more
> > resources, and/or an open source hosted CI service ... we'll
probably
> > have to ignore this con.
>
> I was wrong about Travis. Travis is Open Source. Yay. Good for them.
> That changes how much effort I feel we should dedicate to helping make
> it work, filing bugs, etc.
>
> A little bird also whispered in my ear that Marius has found a
> work-around for the Travis bug that has been bothering us.
>
>
> Not sure if this would help, but the AOS team has a Jenkins server where
> we could do some more sophisticated CI testing:
>
> https://ci.openshift.redhat.com/jenkins/
>
> I am one of the maintainers so I'd be happy to help set things up if
> this was of interest.
Interesting. Have you migrated anything from Travis to that Jenkins
instance? What did the process end up looking like?
I have not done a Travis to Jenkins migration. However, based on looking
at a reference Travis config[1], I'd say the porting is relatively
painless. The Jenkins host needs to have the right tools installed on it
and then you write a shell script for Jenkins to run.
[1]: https://github.com/openshift/origin/blob/master/.travis.yml
But this is actually more interesting for the workloads we don't run on
Travis ... our integration tests ... which launch tons of VMs within the
space of a few minutes. Is that the sort of thing your Jenkins instance
can run? ie: nested virtualization, and/or root access etc?
The public AOS Jenkins system is running on an EC2 instance. With its
current workload, I don't think it has enough power to stand up a bunch
of locally hosted VMs for testing. However, if the tests could be
modified to spin up EC2 instances, or instances on the Red Hat public
OpenStack cluster, then the AOS Jenkins host could certainly handle the
workload of firing them up, running tests, and reporting on the test
results.
True, but we snapshot images, and then spin up VMs, and do a test and
shutdown each within 10 seconds or so. For day to day use (vetting pull
requests) it makes sense to stay on some performant hardware.
Stef
11:51 a.m.
On Mon, Aug 24, 2015 at 9:54 AM, Stef Walter <stefw(a)redhat.com> wrote:
On 24.08.2015 15:44, Harrison Ripps wrote:
>
>
> On Mon, Aug 24, 2015 at 9:27 AM, Stef Walter <stefw(a)redhat.com
> <mailto:stefw@redhat.com>> wrote:
>
> On 24.08.2015 15 <tel:24.08.2015%2015>:15, Harrison Ripps wrote:
> >
> > On Mon, Aug 24, 2015 at 8:45 AM, Stef Walter <stefw(a)redhat.com
<mailto:stefw@redhat.com>
> > <mailto:stefw@redhat.com <mailto:stefw@redhat.com>>> wrote:
> >
> > On 24.08.2015 09 <tel:24.08.2015%2009>
> <tel:24.08.2015%2009>:03, Stef Walter wrote:
> > > On 24.08.2015 08 <tel:24.08.2015%2008>
> <tel:24.08.2015%2008>:54, Marius Vollmer wrote:
> > >> Peter <petervo(a)redhat.com
<mailto:petervo@redhat.com>
> <mailto:petervo@redhat.com <mailto:petervo@redhat.com>>> writes:
> > >>
> > >>> PROS:
> > >>>
> > >>> - It works and uses a newer OS (Ubuntu 14)
> > >>> - Allows you to ssh into instances to run / debug tests.
> > >>
> > >> This is very, very nice.
> > >>
> > >>> CONS:
> > >
> > > Obviously another big con is that this stuff is not Open
Source. These
> > > are proprietary services. But so is GitHub. But until we
have more
> > > resources, and/or an open source hosted CI service ... we'll
probably
> > > have to ignore this con.
> >
> > I was wrong about Travis. Travis is Open Source. Yay. Good for
them.
> > That changes how much effort I feel we should dedicate to
helping make
> > it work, filing bugs, etc.
> >
> > A little bird also whispered in my ear that Marius has found a
> > work-around for the Travis bug that has been bothering us.
> >
> >
> > Not sure if this would help, but the AOS team has a Jenkins server
where
> > we could do some more sophisticated CI testing:
> >
> > https://ci.openshift.redhat.com/jenkins/
> >
> > I am one of the maintainers so I'd be happy to help set things up
if
> > this was of interest.
>
> Interesting. Have you migrated anything from Travis to that Jenkins
> instance? What did the process end up looking like?
>
>
> I have not done a Travis to Jenkins migration. However, based on looking
> at a reference Travis config[1], I'd say the porting is relatively
> painless. The Jenkins host needs to have the right tools installed on it
> and then you write a shell script for Jenkins to run.
>
> [1]: https://github.com/openshift/origin/blob/master/.travis.yml
>
>
> But this is actually more interesting for the workloads we don't run
on
> Travis ... our integration tests ... which launch tons of VMs within
the
> space of a few minutes. Is that the sort of thing your Jenkins
instance
> can run? ie: nested virtualization, and/or root access etc?
>
>
> The public AOS Jenkins system is running on an EC2 instance. With its
> current workload, I don't think it has enough power to stand up a bunch
> of locally hosted VMs for testing. However, if the tests could be
> modified to spin up EC2 instances, or instances on the Red Hat public
> OpenStack cluster, then the AOS Jenkins host could certainly handle the
> workload of firing them up, running tests, and reporting on the test
> results.
True, but we snapshot images, and then spin up VMs, and do a test and
shutdown each within 10 seconds or so. For day to day use (vetting pull
requests) it makes sense to stay on some performant hardware.
Understood. I may still be able to help with this one. I'll follow up after
I've had a chance to talk to our ops team.
Stef
_______________________________________________
cockpit-devel mailing list
cockpit-devel(a)lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/cockpit-devel
1:10 a.m.
Stef Walter <stefw(a)redhat.com> writes:
A little bird also whispered in my ear that Marius has found a
work-around for the Travis bug that has been bothering us.
Yes, see here https://github.com/travis-ci/travis-ci/issues/4704
Since I can't seem to let go of this, I'll probably try to make a simple
reproducer and maybe even dig into the code a bit. (But I have to say I
don't know enough about ptys to be of immediate help here.)
1:42 a.m.
On 25.08.2015 08:10, Marius Vollmer wrote:
Stef Walter <stefw(a)redhat.com> writes:
> A little bird also whispered in my ear that Marius has found a
> work-around for the Travis bug that has been bothering us.
Yes, see here https://github.com/travis-ci/travis-ci/issues/4704
Since I can't seem to let go of this, I'll probably try to make a simple
reproducer and maybe even dig into the code a bit. (But I have to say I
don't know enough about ptys to be of immediate help here.)
For this case here, a PTY acts a lot like a socketpair(). When you
create it you get two file descriptors both readable/writable. The
process is connected to one side, and the other side is read/written by
the terminal (Travis logging system in this case). As you may expect,
there's no concept of stderr in a PTY ... it's all one output stream
(eg: it's all mixed in your terminal).
Stef
3165
days inactive
3169
days old
cockpit-devel@lists.fedorahosted.org
11 comments
4 participants
participants (4)
-
Harrison Ripps -
Marius Vollmer -
Peter -
Stef Walter