Can we use emulation of other architectures to run integration tests?
by Aleksandra Fedorova
Hi, all,
I'd like to get some understanding on the current state of emulation
of other architectures.
In the current CI infra we have infinite(*) access to x86_64 compute
resources, but we haven't yet got our hands on any non x86_64
hardware.
As COPR has recently got support for s390 builds, the question is: if
emulation is good enough for building packages, can we use it for
testing? What are the limitations there? Is it worth it?
If yes, can you suggest some links, tips on the topic?
(*) Not really, and some of our infra services are not yet good at
scaling up, but we are getting better at it.
--
Aleksandra Fedorova
bookwar
3 years, 8 months
Re: We have to talk about annobin... again
by Neal Gompa
On Sat, Jul 25, 2020 at 11:25 AM Jeff Law <law(a)redhat.com> wrote:
>
> On Sat, 2020-07-25 at 10:46 +0100, Peter Robinson wrote:
> > On Sat, Jul 25, 2020 at 10:35 AM Neal Gompa <ngompa13(a)gmail.com> wrote:
> > > Hey all,
> > >
> > > So I was trying to update libseccomp last night, and I was able to
> > > build it for everything except aarch64 on Rawhide because it says the
> > > compiler can't build executables[1].
> > >
> > > Looking a bit closer, it looks like the compiler stack is out of sync
> > > again with annobin.
> > >
> > > Is there anything that can be done to keep the compiler teams from
> > > submitting gcc into rawhide without doing the required rebuild cycle
> > > to make it so annobin works?
> > >
> > > And we're going to have the same problem with clang now that annobin
> > > grew a clang plugin, so I would want neither LLVM nor GCC to land in
> > > Rawhide unless those teams are literally ensuring that annobin isn't
> > > breaking the compiler afterward.
> > >
> > > I'm personally very tired of having the compiler break so frequently
> > > because of that plugin. Either some kind of mechanism to hold back GCC
> > > builds until annobin works is implemented, or I'd much rather see the
> > > whole thing go away. Obviously, you could just *bundle* annobin into
> > > the GCC package and build it together to ensure it never broke, but
> > > that option was discarded already[2].
> > >
> > > Somebody fix it. ASAP.
> >
> > I've kicked off an annobin build for the short term issue.
> >
> > I agree here, we used to have the same issue with gcc-plugins back in
> > the day, sadly this is nothing new :-/
> :( I'll raise it again with Nick and Jakub, it's a sore point for everyone I
> think and it causes far more friction than just what we see here in Fedora.
>
There is now a FESCo ticket proposing all GCC and Clang updates to be
blocked on some kind of gating mechanism:
https://pagure.io/fesco/issue/2454
--
真実はいつも一つ！/ Always, there's only one truth!
3 years, 9 months
We have to talk about annobin... again
by Neal Gompa
Hey all,
So I was trying to update libseccomp last night, and I was able to
build it for everything except aarch64 on Rawhide because it says the
compiler can't build executables[1].
Looking a bit closer, it looks like the compiler stack is out of sync
again with annobin.
Is there anything that can be done to keep the compiler teams from
submitting gcc into rawhide without doing the required rebuild cycle
to make it so annobin works?
And we're going to have the same problem with clang now that annobin
grew a clang plugin, so I would want neither LLVM nor GCC to land in
Rawhide unless those teams are literally ensuring that annobin isn't
breaking the compiler afterward.
I'm personally very tired of having the compiler break so frequently
because of that plugin. Either some kind of mechanism to hold back GCC
builds until annobin works is implemented, or I'd much rather see the
whole thing go away. Obviously, you could just *bundle* annobin into
the GCC package and build it together to ensure it never broke, but
that option was discarded already[2].
Somebody fix it. ASAP.
[1]: https://koji.fedoraproject.org/koji/taskinfo?taskID=47796366
[2]: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.o...
--
真実はいつも一つ！/ Always, there's only one truth!
3 years, 9 months
Re: As a developer I want to easily execute/enable tests
by Petr Šplíchal
Hi Dominik,
Thank you for your questions. See my comments inline.
On Thu, 23 Jul 2020 at 14:39, Dominik 'Rathann' Mierzejewski
<dominik(a)greysector.net> wrote:
>
> On Wednesday, 22 July 2020 at 14:49, Petr Šplíchal wrote:
> > Hi!
> >
> > I am writing today to share a proposal for how we can make it
> > easier to enable tests in Fedora CI.
> >
> > We've heard the user story mentioned in the subject many times.
> > Fedora developers have frequently mentioned the difficulties which
> > they face with enabling tests in Fedora CI. Some people don't
> > understand the complex yaml syntax. Other people find that they
> > can't easily reproduce issues revealed by Fedora CI on their
> > laptops. People have asked us to provide a tool for local testing:
> >
> > https://pagure.io/fedora-ci/general/issue/4
> >
> > A few of us have been working on a tool which we hope will improve
> > the situation considerably. The proof of concept below shows how
> > everyday developer tasks (related to testing) might look in the
> > near future:
> >
> > Enable basic smoke test for my component:
> >
> > dnf install -y tmt
> > tmt init --template mini
> > vim plans/example.fmf
> > git add . && git commit -m tests && git push
> >
> > Create a new test based on a shell/beakerlib template:
> >
> > tmt test create --template shell tests/basic
> > tmt test create --template beakerlib tests/advanced
> >
> > Run all/selected tests safely in a virtual machine:
> >
> > tmt run
> > tmt run test --name smoke
>
> I'd say it's good to have such option, but for quick testing a full VM
> is often too heavy and slow to boot. Even a container takes time to set
> up and run.
Yes, there are different needs and preferences related to the
environment where the tests should be executed. We want to provide
enough flexibility here so the 'provision' step allows you to
choose your preferred way.
> My primary environment would be local.
So for your case that would be 'provision --how local'. If you
have a test box prepared for testing you could also connect to it
using 'provision --how connect'.
> > Execute tests in my preferred environment:
> >
> > tmt run --all provision --how container
> > tmt run --all provision --how virtual
> > tmt run --all provision --how local
> > tmt run --all provision --how ...
>
> I find it difficult to guess what the above commands actually do.
> Do they run the tests or do they provision environments to run the
> tests? Or both?
The 'tmt run' command is used to run tests. By default it runs all
steps (discover, provision, prepare, execute, report, finish) so
the examples above would run tests in the requested environment.
It is also possible to run only selected steps: For example, one
of the use cases shared with us was 'just tell me what tests would
be run, do not run them', this can be achieved with the following:
tmt run discover
For detailed description of individual steps have a look here:
https://tmt.readthedocs.io/en/latest/spec/steps.html
> > I would like to ask for your feedback about this. What do you
> > think? Is this something that would be useful for you? What do
> > you think of the syntax? We would love to hear back from you
> > concerning what essential features you might expect from such a
> > tool. It would also be helpful to know if there is anything that
> > should be changed early in the tool design.
> >
> > For more examples and frequently asked questions have a look here:
> >
> > https://tmt.readthedocs.io/en/latest/examples.html
> > https://docs.fedoraproject.org/en-US/ci/tmt/
> >
> > Looking forward to your feedback and ideas! :)
>
> tmt examples are overcomplicated and not explained in enough detail.
> What is "metadata" and why do you need 4 "levels" of it?
Metadata levels allow to store all needed test execution data at
appropriate location, the levels are described here:
https://tmt.readthedocs.io/en/latest/spec.html
We want the config for simple/common use cases (such as running a
shell script as a test) as simple as possible so that users do not
have to study all the details but just create something like this:
execute:
script: ./run-my-test.sh
If one needs to maintain larger set of tests the L1 metadata can
come handy:
https://tmt.readthedocs.io/en/latest/spec/tests.html
> What are "tiers" for?
They are used to organize tests into groups by priority.
https://tmt.readthedocs.io/en/latest/spec/tests.html#tier
> Which items are mandatory and which are optional?
In order to enable tests you just have to define the 'execute'
step. Use 'tmt init --template mini' to get a quick start.
> What's "relevancy"?
It's a test (L1) attribute allowing to filter relevant tests.
https://tmt.readthedocs.io/en/latest/spec/tests.html#relevancy
Currently we're working on open-sourcing the implementation.
> Why is there "extra-summary" in addition to "summary"?
> What's "extra-task"?
These are because of backward compatibility, for more see:
https://tmt.readthedocs.io/en/latest/stories/cli.html?highlight=extra-tas...
> What are "plans" for?
Plans define which tests should be executed, how the environment
should be provisioned and prepared, how the results should be
reported. See the following page for more details about plans.
https://tmt.readthedocs.io/en/latest/spec/plans.html
> What are "stories" for?
Stories are an advanced, optional feature which can help you to track
which features/requirements have been already implemented, tested
and documented.
https://tmt.readthedocs.io/en/latest/spec/stories.html
https://github.com/psss/tmt/tree/master/stories
> You're using a lot of terms as if everyone knows what they mean
> in your context.
Sorry for that. We're trying to document the concept as best as we
can. Hopefully, the links above have helped. Perhaps we could add
some more references to the examples page.
Thank you for all your feedback. Please, let us know what's still
missing and how we can further improve the docs. Thanks.
psss...
> Regards,
> Dominik
> --
> Fedora https://getfedora.org | RPM Fusion http://rpmfusion.org
> There should be a science of discontent. People need hard times and
> oppression to develop psychic muscles.
> -- from "Collected Sayings of Muad'Dib" by the Princess Irulan
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
3 years, 9 months
As a developer I want to easily execute/enable tests
by Petr Šplíchal
Hi!
I am writing today to share a proposal for how we can make it
easier to enable tests in Fedora CI.
We've heard the user story mentioned in the subject many times.
Fedora developers have frequently mentioned the difficulties which
they face with enabling tests in Fedora CI. Some people don't
understand the complex yaml syntax. Other people find that they
can't easily reproduce issues revealed by Fedora CI on their
laptops. People have asked us to provide a tool for local testing:
https://pagure.io/fedora-ci/general/issue/4
A few of us have been working on a tool which we hope will improve
the situation considerably. The proof of concept below shows how
everyday developer tasks (related to testing) might look in the
near future:
Enable basic smoke test for my component:
dnf install -y tmt
tmt init --template mini
vim plans/example.fmf
git add . && git commit -m tests && git push
Create a new test based on a shell/beakerlib template:
tmt test create --template shell tests/basic
tmt test create --template beakerlib tests/advanced
Run all/selected tests safely in a virtual machine:
tmt run
tmt run test --name smoke
Execute tests in my preferred environment:
tmt run --all provision --how container
tmt run --all provision --how virtual
tmt run --all provision --how local
tmt run --all provision --how ...
I would like to ask for your feedback about this. What do you
think? Is this something that would be useful for you? What do
you think of the syntax? We would love to hear back from you
concerning what essential features you might expect from such a
tool. It would also be helpful to know if there is anything that
should be changed early in the tool design.
For more examples and frequently asked questions have a look here:
https://tmt.readthedocs.io/en/latest/examples.html
https://docs.fedoraproject.org/en-US/ci/tmt/
Looking forward to your feedback and ideas! :)
psss...
3 years, 9 months
