We've run into an issue with travis where adding new make files causes
the tests to fail with an "write error" when running distclean.
For example:
https://travis-ci.org/cockpit-project/cockpit/jobs/76408030.
We haven't been able to reproduce these failures anywhere else. In
addition their new container based infrastructure that they are moving
towards doesn't support sudo which we need for our reauthorize tests. So
we are looking at replacing travis with a different CI service.
I looked at semaphoreci and circleci. I was eventually able to get our
test suite to pass on both systems. Both seems to have most of the same
github integration / badges but Circleci is based on Ubuntu 12 and we'd
really like to be able to use something newer.
You can see my cockpit fork on semaphoreci here:
https://semaphoreci.com/petervo/cockpit
PROS:
- It works and uses a newer OS (Ubuntu 14)
- Allows you to ssh into instances to run / debug tests.
CONS:
- There is only one configuration per project that is configured via
their website. So you can't change the commands on a per branch / PR
basis. Additionally the webui has some bugs and usability issues that
end up requiring a few tries before it gets the commands in the right
order and in the right threads.
- We have to run on their container based beta platform as their normal
platform doesn't fully support inotify and causes tests to fail.
- Doesn't do IRC notifications out of the box. Though we could probably
implement something using webhooks if we wanted to.
What does everyone think about switching? Worth doing or should we keep
looking at other options?