Recipe Parsing Issue
by Radek Pazdera
Hi,
there is one more issue I wanted to discuss with you guys, but
I completely forgot to mention it at the meeting we had on Monday.
It's the thing with parser implementation that came up after I finished
writing libteam support. The problem is that the way the parsing is
implemented now allows using template funcions (e.g. {ip(1,1)}) only
within a command sequence. The functions can be only used after
all machine and net configs were parsed.
This makes the libteam implementation practically unusable. It works
correctly, but you have to enter the device names of slave interfaces
to the teamd config (which is contained within netconfig) by hand.
We discussed it a while ago with Jirka and it seems, that there's no
easy way around this :-/. The solution is to make the parser work on
a line-by-line basis and process actions associated with certain lines
right away. Now things are a little optimized, so the controller sends
only one big request to each slave with the whole netconfig, which
results in this not-very-intuitive behavior of recipes.
I personally think, that this could really simplify conditions of using
the template functions and aliases from a obscure set of rules (that
can be hard to get for people not involved int LNST development) to
a simple rule "you can use whatever you see written above this
line".
Do you think this is worth the work at the moment or I should
leave it for now and get back to it later?
I don't know if my explanation is clear enough ... I can explain it to
you in more detail in person sometime.
Radek
11 years, 10 months
Dropping recipe_eval
by Radek Pazdera
What is your opinion on keeping recipe_eval options
in LNST? The same functionality can be now accessed
using a predefined template variable $recipe. For instance:
{$recipe["machines"][1]["info"]["hostname"]}
So the older way is not needed any more.In my opinion, it
would be better to stick to one way of accessing the recipe
to maintain consistency among the recipes, but dropping
recipe_eval entirely, could break some older ones.
What do you think? We could either drop it entirely or
keep it in as "undocumented".
Radek
11 years, 10 months
Native packet capturing for LNST
by Radek Pazdera
Hi,
I started working on the packet assert test this week, that we discussed
earlier
and we discovered, that the idea of packet capturing might have much broader
use in LNST.
Jirka had an idea, that we could put an optional parameter to the
controler to
enable debugging mode, in which communication on all interfaces would be
captured by tcpdump, logged and then transfered back to controller along
with the logs.
This could also be useful along with tcpreplay to catch and reproduce some
particularly sneaky bugs. tcpreplay can be run as an exec from command
sequence to simulate previous situation within the network in later recipe
runs.
I tried to figure out a way to implement this efficiently, but I can't
really come
up with a straight-forward solution at the moment. We could use tcpdump
to create a *.pcap dump file for each interface and then analyze it with
capinfo (it's a part of wireshark) or again with tcpdump to get some
information.
Or there is a option to bypass tcpdump and use libpcap directly through some
of it's python bindings (pypcap, pcapy or pylibpcap) along with dkpt
library for
parsing packets, but I'm not sure what's the current state of development of
these things and how stable they are.
I guess we could start simple with tcpdump and optimize it later in case
some
performance issue comes up.
I also looked at tcpdump's code at how they parse the packets' headers into
strings and, they have separate C modules for every supported protocol type
(called print-<protocol_name>.c), so they are not using any library that
we could
use easily.
Then there's a native capture tool for python called "scapy", but it's
supposed
to be very slow and buggy (as of what I read about it, I haven't tried
running it).
The interface in LNST then could look like this:
<netconfig>
<netdevice id="1" type="eth" phys_id="1">
<addresses>
<address value="192.168.122.225/24" />
</addresses>
<capture enabled="true">
<options>
<option name="filter" value="tcp" />
<option name="format" value="binary" />
<option name="keep_log" value="true" />
<option name="stats" value="true" />
</options>
</capture>
</netdevice>
</netconfig>
This config would enable capturing with custom filter to get only tcp
traffic, it would keep
the log when the recipe is done and transfer it back to controller (the
log option) and
controller then would read all the files with capinfo and printed some
stats.
At first, I thought that the stats should be gathered periodically as
the packets come
in, but it's rather ineffective to bring each packet through an
additional process, that
would have to run and count them, when we can do this later in
postprocessing
where performance is not an issue anymore.
What do you think? Is this reasonable?
Radek
11 years, 10 months
[lnst] Parsing error fix for short test times
by Jiří Pírko
commit 812ef52dc6f50ba1000a3b2b48de095e6820c657
Author: Jan Tluka <jtluka(a)redhat.com>
Date: Mon Jun 11 15:23:50 2012 +0200
Parsing error fix for short test times
Minor fix for the iperf output parsing.
Case 1:
[ 5] 0.0-10.0 sec 5.21 GBytes 4.47 Gbits/sec
Case 2:
[ 5] 0.0- 5.0 sec 3.52 GBytes 6.04 Gbits/sec
In case #2 the space after dash was not expected and resulted in
parsing error. The fix corrects this and handles both cases.
Tests/TestIperf.py | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
---
diff --git a/Tests/TestIperf.py b/Tests/TestIperf.py
index 9b67d40..f6ba232 100644
--- a/Tests/TestIperf.py
+++ b/Tests/TestIperf.py
@@ -91,7 +91,7 @@ class TestIperf(TestGeneric):
output = client.read_nonblocking()
if self.threshold is not None:
# check if expected threshold is reached
- m = re.search("\[[^0-9]*[0-9]*\]\s*0.0-\d*\.\d sec\s*\d*(\.\d*){0,1}\s*[ kGMT]Bytes\s*(\d*(\.\d*){0,1}\s*[ kGMT]bits\/sec)", output)
+ m = re.search("\[[^0-9]*[0-9]*\]\s*0.0-\s*\d*\.\d sec\s*\d*(\.\d*){0,1}\s*[ kGMT]Bytes\s*(\d*(\.\d*){0,1}\s*[ kGMT]bits\/sec)", output)
if m is None:
logging.info("Could not get performance throughput!")
return False
11 years, 10 months
[lnst] Fix the wrong parsing of throughput value
by Jiří Pírko
commit 3ebbc12d5ad1a35cdd03c8f9645ebeb3acf61793
Author: Jan Tluka <jtluka(a)redhat.com>
Date: Mon Jun 11 14:25:48 2012 +0200
Fix the wrong parsing of throughput value
Tests/TestIperf.py | 10 ++--------
1 files changed, 2 insertions(+), 8 deletions(-)
---
diff --git a/Tests/TestIperf.py b/Tests/TestIperf.py
index cfea973..9b67d40 100644
--- a/Tests/TestIperf.py
+++ b/Tests/TestIperf.py
@@ -49,15 +49,9 @@ class TestIperf(TestGeneric):
r2 = re.match(pattern, rate)
rate_units = r2.group(3)
- if r1.group(2) != None:
- thr_val = float(r1.group(1) + r1.group(2))
- else:
- thr_val = float(r1.group(1))
+ thr_val = float(r1.group(1))
- if r2.group(2) != None:
- rate_val = float(r2.group(1) + r2.group(2))
- else:
- rate_val = float(r2.group(1))
+ rate_val = float(r2.group(1))
# do the conversion of rate units
if thr_units != rate_units:
11 years, 10 months
[lnst] NetConfig: Adding libteam/teamdev support
by Jiří Pírko
commit 1a487a1bbda9a5a5a0bcfd8cc9d95decb0847ed2
Author: Radek Pazdera <rpazdera(a)redhat.com>
Date: Fri Jun 8 12:19:28 2012 +0200
NetConfig: Adding libteam/teamdev support
This commit introduces team device support to LNST. Example net
configuration to demonstrate usage of team with LNST was added as well.
Signed-off-by: Radek Pazdera <rpazdera(a)redhat.com>
NetConfig/NetConfigDevNames.py | 6 ++++
NetConfig/NetConfigDevice.py | 36 +++++++++++++++++++++++++++-
NetConfig/NetConfigParse.py | 2 +-
example_recipes/net_configs/team1.xml | 42 +++++++++++++++++++++++++++++++++
4 files changed, 84 insertions(+), 2 deletions(-)
---
diff --git a/NetConfig/NetConfigDevNames.py b/NetConfig/NetConfigDevNames.py
index a8636f6..8522e00 100644
--- a/NetConfig/NetConfigDevNames.py
+++ b/NetConfig/NetConfigDevNames.py
@@ -76,6 +76,9 @@ class NetConfigDevNames:
def _assign_name_macvlan(self, netdev, config):
self._assign_name_generic("t_macvlan", netdev, config)
+ def _assign_name_team(self, netdev, config):
+ self._assign_name_generic("t_team", netdev, config)
+
def _assign_name_vlan(self, netdev, config):
real_netdev = config[netdev["slaves"][0]]
vlan_tci = get_option(netdev, "vlan_tci")
@@ -95,6 +98,9 @@ class NetConfigDevNames:
self._assign_name_bridge(netdev, config)
elif dev_type == "macvlan":
self._assign_name_macvlan(netdev, config)
+ elif dev_type == "team":
+ self._assign_name_team(netdev, config)
+
'''
In second round assign names for vlans as they use
previously assigned names
diff --git a/NetConfig/NetConfigDevice.py b/NetConfig/NetConfigDevice.py
index e024db7..be8e673 100644
--- a/NetConfig/NetConfigDevice.py
+++ b/NetConfig/NetConfigDevice.py
@@ -212,12 +212,46 @@ class NetConfigDeviceVlan(NetConfigDeviceGeneric):
else:
exec_cmd("vconfig rem %s" % dev_name)
+class NetConfigDeviceTeam(NetConfigDeviceGeneric):
+ _pidfile = None
+
+ def _slaves_up(self):
+ for slaveid in get_slaves(self._netdev):
+ slavenetdev = self._config[slaveid]
+ NetConfigDevice(slavenetdev, self._config).up()
+
+ def _slaves_down(self):
+ for slaveid in get_slaves(self._netdev):
+ slavenetdev = self._config[slaveid]
+ NetConfigDevice(slavenetdev, self._config).down()
+
+ def configure(self):
+ self._slaves_down()
+
+ teamd_config = get_option(self._netdev, "teamd_config")
+ teamd_config = teamd_config.replace('"', '\\"')
+
+ dev_name = self._netdev["name"]
+ pidfile = "/var/run/teamd_%s.pid" % dev_name
+
+ exec_cmd("teamd -r -d -c \"%s\" -t %s -p %s" % (teamd_config, dev_name, pidfile))
+
+ self._pidfile = pidfile
+
+ def deconfigure(self):
+ dev_name = self._netdev["name"]
+ pidfile = "/var/run/teamd_%s.pid" % dev_name
+
+ exec_cmd("teamd -k -p %s" % pidfile)
+ self._slaves_up()
+
type_class_mapping = {
"eth": NetConfigDeviceEth,
"bond": NetConfigDeviceBond,
"bridge": NetConfigDeviceBridge,
"macvlan": NetConfigDeviceMacvlan,
- "vlan": NetConfigDeviceVlan
+ "vlan": NetConfigDeviceVlan,
+ "team": NetConfigDeviceTeam
}
def NetConfigDevice(netdev, config):
diff --git a/NetConfig/NetConfigParse.py b/NetConfig/NetConfigParse.py
index ff148e5..9097e88 100644
--- a/NetConfig/NetConfigParse.py
+++ b/NetConfig/NetConfigParse.py
@@ -192,7 +192,7 @@ class NetConfigParse:
if dev_type == "eth":
pass
- elif dev_type in ["bond", "bridge", "vlan", "macvlan"]:
+ elif dev_type in ["bond", "bridge", "vlan", "macvlan", "team"]:
self._parse_options(netdev, dom_netdev, config)
self._parse_slaves(netdev, dom_netdev, config)
else:
diff --git a/example_recipes/net_configs/team1.xml b/example_recipes/net_configs/team1.xml
new file mode 100644
index 0000000..4bebf13
--- /dev/null
+++ b/example_recipes/net_configs/team1.xml
@@ -0,0 +1,42 @@
+<netconfig>
+ <netdevice id="1" type="eth" phys_id="1" />
+ <netdevice id="2" type="eth" phys_id="2" />
+ <netdevice id="5" type="team">
+ <options>
+ <!-- Team device name can be omitted from teamd config,
+ because it will be overridden by LNST anyway. -->
+ <option name="teamd_config">
+ {
+ "runner": {"name": "roundrobin"},
+ "ports": {"eth1": {}, "eth2": {}}
+ }
+ </option>
+ </options>
+ <slaves>
+ <slave id="1" />
+ <slave id="2" />
+ </slaves>
+ <addresses>
+ <address value="192.168.122.200/24" />
+ </addresses>
+ </netdevice>
+ <netdevice id="3" type="eth" phys_id="3" />
+ <netdevice id="4" type="eth" phys_id="4" />
+ <netdevice id="6" type="team">
+ <options>
+ <option name="teamd_config">
+ {
+ "runner": {"name": "roundrobin"},
+ "ports": {"eth3": {}, "eth4": {}}
+ }
+ </option>
+ </options>
+ <slaves>
+ <slave id="3" />
+ <slave id="4" />
+ </slaves>
+ <addresses>
+ <address value="192.168.122.210/24" />
+ </addresses>
+ </netdevice>
+</netconfig>
11 years, 10 months
[lnst] NetTestParse: Remove XML comments before parsing
by Jiří Pírko
commit 9752c3399e6e1d947c9542f8801c0db24ca97f85
Author: Radek Pazdera <rpazdera(a)redhat.com>
Date: Fri Jun 8 12:19:27 2012 +0200
NetTestParse: Remove XML comments before parsing
Comments in the XML recipe might interfere with the parser in some
cases and result in exceptions. The best way to deal with this is
to remove them from the recipe.
Signed-off-by: Radek Pazdera <rpazdera(a)redhat.com>
Common/XmlPreprocessor.py | 15 +++++++++++++++
NetTest/NetTestParse.py | 2 ++
2 files changed, 17 insertions(+), 0 deletions(-)
---
diff --git a/Common/XmlPreprocessor.py b/Common/XmlPreprocessor.py
index 88ae758..b20fec8 100644
--- a/Common/XmlPreprocessor.py
+++ b/Common/XmlPreprocessor.py
@@ -51,6 +51,21 @@ class XmlPreprocessor:
else:
raise XmlTemplateError("Alias name '%s' is reserved" % name)
+ def remove_comments(self, node):
+ """
+ Remove all comment nodes from the tree.
+ """
+
+ comments = []
+ for child in node.childNodes:
+ if child.nodeType == node.COMMENT_NODE:
+ comments.append(child)
+ else:
+ self.remove_comments(child)
+
+ for comment in comments:
+ node.removeChild(comment)
+
def expand(self, node):
"""
Traverse DOM tree from `node' down and expand any
diff --git a/NetTest/NetTestParse.py b/NetTest/NetTestParse.py
index 5b5101a..68c0058 100644
--- a/NetTest/NetTestParse.py
+++ b/NetTest/NetTestParse.py
@@ -81,6 +81,8 @@ class NetTestParse:
dom = parseString(self._recipe_xml_string)
xml_prep = self._xml_prep
+ xml_prep.remove_comments(dom)
+
self._load_included_parts(dom)
dom_nettestrecipe = dom.getElementsByTagName("nettestrecipe")[0]
11 years, 10 months
[lnst] NetConfigParse: Extending options parsing
by Jiří Pírko
commit cefbcea3515f7010c955aa5afe783e96d108ea4c
Author: Radek Pazdera <rpazdera(a)redhat.com>
Date: Fri Jun 8 12:19:25 2012 +0200
NetConfigParse: Extending options parsing
This commit makes NetConfigParser accept <netdevice> options in a
slightly different format. Previous format of options accepted values
only as an 'value' attribute to the <option> tag. With this modification
it is possible to define options also with values as a text elements of
the option tag. There are now two possibilities:
<!-- The usual way -->
<option name="opt" value="5" />
<!-- Longer values can be specified like this -->
<option name="teamd_config">
{
"device": "team1",
"runner": {"name": "roundrobin"},
"ports": {"eth3": {}, "eth4": {}}
}
</option>
Signed-off-by: Radek Pazdera <rpazdera(a)redhat.com>
NetConfig/NetConfigParse.py | 12 +++++++++++-
1 files changed, 11 insertions(+), 1 deletions(-)
---
diff --git a/NetConfig/NetConfigParse.py b/NetConfig/NetConfigParse.py
index dbd7a6d..ff148e5 100644
--- a/NetConfig/NetConfigParse.py
+++ b/NetConfig/NetConfigParse.py
@@ -97,7 +97,17 @@ class NetConfigParse:
def _parse_options_handler(self, lst, dom_element, config):
name = str(dom_element.getAttribute("name"))
- value = str(dom_element.getAttribute("value"))
+
+ value = ""
+ if dom_element.hasAttribute("value"):
+ value = str(dom_element.getAttribute("value"))
+ elif dom_element.hasChildNodes():
+ node = dom_element.firstChild
+ try:
+ value = str(dom_element.firstChild.data)
+ except:
+ raise Exception("Invalid option value")
+
lst.append((name, value))
def _parse_options(self, netdev, dom_netdev, config):
11 years, 10 months
[lnst] Exclude tarball and git directory from generated tarball
by Jiří Pírko
commit 3ceb5e8d837997fbbb4e77d5fb6d74437f9522a8
Author: Jan Tluka <jtluka(a)redhat.com>
Date: Wed Jun 6 17:06:17 2012 +0200
Exclude tarball and git directory from generated tarball
Make tarball smaller excluding git metadata directory and possibly previously
generated tarball.
Common/SlaveUtils.py | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
---
diff --git a/Common/SlaveUtils.py b/Common/SlaveUtils.py
index 3640f94..2c1f043 100644
--- a/Common/SlaveUtils.py
+++ b/Common/SlaveUtils.py
@@ -27,7 +27,7 @@ def prepare_client_session(host, port, login, passwd=None, command=None,
if test_dir is None:
test_dir = "lnst"
- s = ShellProcess("tar -cjf lnst.tar.bz2 --exclude *.pyc --exclude 'Logs/*' -C '%s' ./" % sys.path[0])
+ s = ShellProcess("tar -cjf lnst.tar.bz2 --exclude *.pyc --exclude 'Logs/*' --exclude '.git/*' --exclude 'lnst.tar.bz2' -C '%s' ./" % sys.path[0])
s.wait()
scp_to_remote(host, port, login, passwd,
"lnst.tar.bz2","/%s/" % (install_path))
11 years, 10 months
[lnst] Make remote_exec working if executed outside of lnst root directory
by Jiří Pírko
commit 68063fcbe8c3f120eaa822e059187bcbb60fe88e
Author: Jan Tluka <jtluka(a)redhat.com>
Date: Wed Jun 6 17:04:34 2012 +0200
Make remote_exec working if executed outside of lnst root directory
If controller is executed outside of LNST source directory the generated
tarball contains content of current working directory instead of the LNST
code. The fix is to change the current directory using -C tar option and
use of ./ instead of * wildcard.
Common/SlaveUtils.py | 7 ++++---
1 files changed, 4 insertions(+), 3 deletions(-)
---
diff --git a/Common/SlaveUtils.py b/Common/SlaveUtils.py
index e7fdf3d..3640f94 100644
--- a/Common/SlaveUtils.py
+++ b/Common/SlaveUtils.py
@@ -8,7 +8,7 @@ jzupka(a)redhat.com (Jiri Zupka)
from Common.SshUtils import scp_to_remote, wait_for_login
from Common.ShellProcess import ShellProcess
-
+import sys
def prepare_client_session(host, port, login, passwd=None, command=None,
prompt=None, install_path=None, test_dir=None):
@@ -26,7 +26,8 @@ def prepare_client_session(host, port, login, passwd=None, command=None,
install_path = "/tmp"
if test_dir is None:
test_dir = "lnst"
- s = ShellProcess("tar -cjf lnst.tar.bz2 --exclude *.pyc --exclude 'Logs/*' *")
+
+ s = ShellProcess("tar -cjf lnst.tar.bz2 --exclude *.pyc --exclude 'Logs/*' -C '%s' ./" % sys.path[0])
s.wait()
scp_to_remote(host, port, login, passwd,
"lnst.tar.bz2","/%s/" % (install_path))
@@ -39,4 +40,4 @@ def prepare_client_session(host, port, login, passwd=None, command=None,
prompt = "Started"
command = "cd /%s/%s/ && ./%s" % (install_path, test_dir, command)
return wait_for_login(host, port, login, passwd, prompt,
- command=command, timeout=10)
\ No newline at end of file
+ command=command, timeout=10)
11 years, 10 months