Hello,
I've some tests written in Python (all under 50 lines), used to test for the following software:
Dovecot : 1 test Apache HTTPD: 3 + 1 not yet finished vsFTPd : 1 test MySQL : 1 test SSHD : 1 not yet finished
After reading about autotest (big picture) and 'Writing AutoQA Tests' [1], I need to have the following info:
1. Should I use TEST_TYPE = 'CLIENT', because actually the tests run on 'localhost' or I need to modify the tests to be network-aware and use TEST_TYPE = 'SERVER'.
2. I'm little confused about the directory structure, should I group all tests into one or split the according the service (tests/httpd, tests/httpd_vhost, tests/vsftpd, etc...)
3. The tests uses some no-standards Python modules like pexpect and 'tcpcat' (I wrote this module to emulate netcat, because tests have failed after a netcat bug[2] ), Is that OK ?
[1] https://fedoraproject.org/wiki/Writing_AutoQA_Tests [2] https://bugzilla.redhat.com/show_bug.cgi?id=695108
Best regards.
On Mon, 2011-04-18 at 06:54 +0100, Athmane Madjoudj wrote:
Hello,
I've some tests written in Python (all under 50 lines), used to test for the following software:
Wow, this is great! First, let me say thanks for being proactive and attempting to create AutoQA tests or your packages. Presently, AutoQA is focused on tests that apply to all packages (e.g. rpmlint, depcheck, upgradepath etc...). At some point in the future, AutoQA will support package-specific tests contributed by maintainers. However, at this time, we are not able to offer that support.
The main roadblocks I'm aware of are crafting a solution to provide disposable test systems. With disposable test systems, each test would get a fresh new system to test on, and wouldn't need to worry about destructive tests or test cleanup. Another roadblock is defining a structure for where and how maintainers will write tests. I don't think we need to expose all of the autotest/autoqa library to maintainers, it's just too much information. We need to figure out what information is needed, and provide a simple way for maintainers to store tests such that AutoQA will run them at a desired event.
We could always use help building the infrastructure for package-specific tests. But in the meantime, the options for creating and running your tests are probably ...
1. Work with upstream to have your tests included in any upstream test suite 2. Include your tests in the rpm and have them run by %check during package build 3. Create a private [1] autoqa git branch that mirrors 'stable' and manually run your tests. You would be responsible for running the tests and communicating the results, but this would at least help you identify issues with integrating tests, and perhaps help expedite the process of package-specific tests.
[1] https://fedoraproject.org/wiki/Infrastructure/fedorapeople.org#Creating_a_ne...
Dovecot : 1 test Apache HTTPD: 3 + 1 not yet finished vsFTPd : 1 test MySQL : 1 test SSHD : 1 not yet finished
After reading about autotest (big picture) and 'Writing AutoQA Tests' [1], I need to have the following info:
- Should I use TEST_TYPE = 'CLIENT', because actually the tests run on
'localhost' or I need to modify the tests to be network-aware and use TEST_TYPE = 'SERVER'.
You'll want CLIENT, unless your tests require multiple systems coordinating with each other using barriers.
- I'm little confused about the directory structure, should I group all
tests into one or split the according the service (tests/httpd, tests/httpd_vhost, tests/vsftpd, etc...)
Since we don't yet support package-specific maintainer contributed tests, the directory structure isn't as important. I'd probably suggest a single directory for each src.rpm ... but that's subject to change and just a best guess at this point.
- The tests uses some no-standards Python modules like pexpect and
'tcpcat' (I wrote this module to emulate netcat, because tests have failed after a netcat bug[2] ), Is that OK ?
The dependencies of the test can be installed using the setup() method in the Autotest wrapper. If your dependencies are not packaged as rpms, they'll need to be, or they can be included in your test. The ideal scenario, is packaging them though.
[1] https://fedoraproject.org/wiki/Writing_AutoQA_Tests [2] https://bugzilla.redhat.com/show_bug.cgi?id=695108
Thanks, James
On 04/18/2011 02:44 PM, James Laska wrote:
On Mon, 2011-04-18 at 06:54 +0100, Athmane Madjoudj wrote:
Hello,
I've some tests written in Python (all under 50 lines), used to test for the following software:
[... SNIP ...]
We could always use help building the infrastructure for package-specific tests. But in the meantime, the options for creating and running your tests are probably ...
1. Work with upstream to have your tests included in any upstream test suite 2. Include your tests in the rpm and have them run by %check during package build 3. Create a private [1] autoqa git branch that mirrors 'stable' and manually run your tests. You would be responsible for running the tests and communicating the results, but this would at least help you identify issues with integrating tests, and perhaps help expedite the process of package-specific tests.[1] https://fedoraproject.org/wiki/Infrastructure/fedorapeople.org#Creating_a_ne...
Thank you for your answers,
All tests are very fedora specific, they add users, modify configuration files (which is distro specific), add entry in /etc/hosts, etc..., IMHO option 1 is not very adequate.
However option 2 is more interesting and doesn't require modification of the current autoqa logic, but those tests requires package to be installed and during %check the package is not.
For example the Dovecot IMAP test:
---8<-------------------------------------------------------------------- #!/usr/bin/python # Author: Athmane Madjoudj athmanem@gmail.com
import os from lib.tcpcat import tcpcat
def setup(): os.system('yum -y install dovecot') if os.system('service dovecot status') != 0: os.system('service dovecot start')
def run(): dovecot_mail = open("/etc/dovecot/conf.d/11-mail.conf","w") dovecot_mail.write("mail_location = mbox:~/mail:INBOX=/var/mail/%u\n") dovecot_mail.write("mail_privileged_group = mail\n") dovecot_mail.close() os.system('service dovecot restart') os.system("useradd imaptest && echo imaptest | passwd --stdin imaptest") print "Dovecot IMAP login test: ", result = tcpcat("localhost",143,'01 LOGIN imaptest imaptest\n') if "Logged in" in result: print "PASS" return 0 else: print "FAIL" return 1
def cleanup(): os.system("userdel -f imaptest") os.system("rm -f /etc/dovecot/conf.d/11-mail.conf")
if __name__ == '__main__': setup() run() cleanup()
Best regards.
On 04/20/2011 10:42 AM, Athmane Madjoudj wrote:
On 04/18/2011 02:44 PM, James Laska wrote:
On Mon, 2011-04-18 at 06:54 +0100, Athmane Madjoudj wrote:
Hello,
I've some tests written in Python (all under 50 lines), used to test for the following software:
[... SNIP ...]
I think I'll use initscripts tests as reference, I've found many similarities with what I want, like:
- Installing the package to be tested. - Add test users - Manipulate service startup etc...
On Thu, 2011-04-21 at 05:05 +0100, Athmane Madjoudj wrote:
On 04/20/2011 10:42 AM, Athmane Madjoudj wrote:
On 04/18/2011 02:44 PM, James Laska wrote:
On Mon, 2011-04-18 at 06:54 +0100, Athmane Madjoudj wrote:
Hello,
I've some tests written in Python (all under 50 lines), used to test for the following software:
[... SNIP ...]
I think I'll use initscripts tests as reference, I've found many similarities with what I want, like:
- Installing the package to be tested.
- Add test users
- Manipulate service startup
etc...
Excellent. Just remember that Fedora 15 (and newer) makes use of systemd rather than the classic SysV initscripts. The good news, I believe using 'chkconfig' and 'service' will remain functional in both worlds.
Are you planning to develop your tests in a git clone [1] that mirrors stable?
Thanks, James
[1] https://fedoraproject.org/wiki/Infrastructure/fedorapeople.org#Creating_a_ne...
On 04/21/2011 03:31 PM, James Laska wrote:
On Thu, 2011-04-21 at 05:05 +0100, Athmane Madjoudj wrote:
On 04/20/2011 10:42 AM, Athmane Madjoudj wrote:
On 04/18/2011 02:44 PM, James Laska wrote:
On Mon, 2011-04-18 at 06:54 +0100, Athmane Madjoudj wrote:
Hello,
Excellent. Just remember that Fedora 15 (and newer) makes use of systemd rather than the classic SysV initscripts. The good news, I believe using 'chkconfig' and 'service' will remain functional in both worlds.
I already have tested those script on fedora 15 (and systemd was a motivation).
Are you planning to develop your tests in a git clone [1] that mirrors stable?
Actually, I'll try to get deeper into AutoQA source, in meantime I continue to write new tests.
autoqa-devel@lists.fedorahosted.org
-
Athmane Madjoudj -
James Laska