On 08/29/2012 01:58 PM, Olaf Kirch wrote: Why Did You Do This?! Do we really need yet another network management thing? ======================================================= No, not really. We already have the good old ifup/ifdown scripts, which are doing their job nicely. *** No they don't. If you don't strain them too much or try to teach them new tricks (like integrating with systemd). Of all the tunables and knobs the kernel supports for each network interface, we're covering maybe 10%, but what was good enough for grandpa should be good enough for me as well, right? We've got udev messing around with network interface names, because users don't like it when what's called eth0 today is called eth15 on the next boot. Which works nicely except when it fails, for one or the other weird reason. We've got Network Manager, which is also doing it's job nicely and won't give you any headaches if you prevent it from stepping on anybody else's toes. Or try to make it manage a thousand devices, like on System z. *** Network Manager is just another daemon created for a task which historically often did not need any daemons. It's almost as if the new generation of Unix hackers wants to redo everything - in x10 or x100 times bloated and more complex way than it was done before. Yes, guys, old stuff has problems and limitations. But switching into high gear and writing a ton of stuff is not a solution I like. Can you *think it through* and fix/improve stuff WITHOUT increasing its complexity tenfold as a minimum? Yes, I do realize that elegant solution is harder to do than coding spree solution, but I still hope... Then we've got libvirt and netcf, which do kind of an okay job if you manage to frob netcf enough that it deals with configuration files other than RedHat's, and as long as your network configuration doesn't get too complicated. Which happens quickly in a virtualization environment. Beyond these, there are things like openvswitch, which is crucial in a cloud environment but not at all integrated with any of the other components. So no, we don't really need yet another network management thingy. We need a management thingy that replaces a lot of this stuff. Yeah, but it kind of works, why should we mess with it ====================================================== Quick, using any of the existing frameworks, can you tell me how to... - ... disable IPv6 on a specific interface? - ... set up an interface for DHCPv4 and DHCPv6? - ... change the link speed on an Ethernet interface? - ... reconfigure a bonding device without bringing it down? - ... set up a bridge using two bonded NICs as one of its ports? - ... the same as above, with VLAN tagging? - ... change the firewall rules on your UMTS modem? - ... set up 802.1x authentication for your Ethernet NIC? - ... set up persistent names for your System z devices? If you could answer all of them at the snap of a finger, please send me your CV. *** I would rather send you a few URLs: http://busybox.net/~vda/no_ifup.txt http://git.busybox.net/busybox/tree/examples/var_service/README So, what properties should a new network management framework have? =================================================================== Obviously, there are a number of aspects of the existing systems that make them useful to people; retaining these is worthwhile. For instance, having a command line interface is crucial to many people. You need to be able to change your network settings from an ssh session; and you want to be able to script network management tasks, too. Similarly, people like NetworkManager because it allows them to set up network interfaces in response to various events, and it does so more or less automatically, and without the user having to be root. too. While shell scripts are certainly not the latest in Software Engineering, *** shell scripts are time tested tools. Before you disparage them, collect at least ten years of successful usage under your tool's belt. it is certainly easy to extend an existing script (provided you understand what it's doing), and to insert debug output, etc. *** exactly. Do the same to XML? I don't think so. Beyond such aspects, there are a number of desirable properties. One, a modern network management framework should run as a service. The kernel offers a plethora of notifications via rtnetlink, and increasingly expects user space to react to these (for instance in the IPv6 area). Running a network management daemon allows us to track the state, detect changes, and react to them appropriately. Two, a modern network management framework should be layered - both in its implementation, and its configuration. One of the reasons why the existing ifcfg files are such a mess are the inherent limitations of shell scripts and shell variables dealing with complex types. BOOTPROTO='static' STARTMODE='auto' NAME='82566DM-2 Gigabit Network Connection' ETHTOOL_OPTIONS='' USERCONTROL='no' IPADDR='1.2.3.4/24' IPADDR_0='1.2.8.17/24' This mixes - in random order - address configuration, hardware configuration, behavior control, and UI information. And things will become worse the moment you start to support additional parameters for your network devices - because you will quickly end up with VERY_LONG_VARIABLE_NAMES_NOBODY_WILL_EVER_REMEMBER. *** You honestly think XML is more readable? If not, then this particular point about VERY_LONG_VARIABLE_NAMES_NOBODY_WILL_EVER_REMEMBER is invalid. But yes, the general point stands. And some things are just tough to do using shell variables (while still maintaining a certain level of sanity), so that auxiliary file formats had to be invented - viz the ifcfg-routes file in opensuse. Ideally, the configuration file format should structure options into logical units - having syntactic groups for lumps of data that belong together; being able to organize several instances of the same type of data (such as say a static route) into a list; etc. Layering is also crucial for the implementation, because it makes sure you support a uniform set of features across all types of devices. Today, setting up a firewall on a serial PPP device is very different from doing it on an Ethernet interface, and that again is very different from doing it on a UMTS PPP session started by NetworkManager. Three, a modern network management framework should support a way to identify network devices by means other than their name. That name is really secondary; and tools should not rely on it. Instead, the management framework should provide naming facilities that allow you to identify interfaces by a set of attributes - for instance, it should be possible to identify a UMTS stick by the IMEI of its GSM card. Or it should be possible to identify a hotplug Ethernet card by the PCI ID of its enclosure, which ensures that when you replace the card, the new one will receive the same configuration as the old one. Four, a modern network management framework should model interface dependencies. Consider a bridge sitting on top of a VLAN built on a pair of bonded NICs. Before bringing up any of these, the management framework should bring up the lower-layer device. This doesn't really happen with today's scripts. Five, a modern network management framework should provide triggers for all sorts of things. For instance, if you have an NFS mount, wouldn't it be nice if you could tell the management framework to notify you as soon as the server's host name can be resolved, and the address is known to be reachable? Why don't you use tool XYZ? =========================== We looked at all of the available tools, but we didn't come across anything that had similar goals to the ones above. Some went a long way, but they were either very much focused on a desktop-like use case with few devices (mostly Ethernet, WLAN and UMTS), or they were focused on the Enterprise end of the spectrum with little support for the needs of the single end user. So, I started to work on something I called wicked initially, which was a REST based service. If you haven't guess so far, I have a passion for really cheesy puns - REST for the wicked was something I couldn't pass up. The thing evolved quite a bit over time, moving from REST to a dbus based transport, among others - and it keeps evolving. Among other things, it'll probably change its name in the not too distant future, because people keep confusing wicked with WICD. I just need to come up with another cheesy pun. Basic architecture ================== The basic architecture of the whole service is rather simple. There's a daemon process, wickedd, passively monitoring all interfaces, without touching any of them unless told otherwise. It offers a view of these devices via DBus, with a number of DBus interfaces attached to each of them. Clients can talk to this service and request a specific operation on any such device. *** How do you set up network parameters? IOW: does wickedd reimplement such tools like ip, ethtool, iwconfig, dhcpc client, ppp? *** Can you show the ldd and size output on your wickedd binary? There is a command line client called wicked whose main purpose it is to act as a backend to ifup and ifdown. This client can configure any number of interfaces in parallel, and also takes care of dependencies between interfaces. There is another application called network-nanny, which tries to do what NetworkManager is currently doing, using the Wicked service. It is fairly tightly integrated with the other components of management framework, and shares a significant amount of code with the command line client. The same, but showing the layering at work: <interface> <name>eth0</name> <control> <mode>boot</mode> <link-detection> <timeout>60</timeout> <require-link /> </link-detection> </control> <ethernet> <port-type>tp</port-type> <link-speed>1000</link-speed> <offload> <tso>disable</tso> </offload> </ethernet> <link> <mtu>8000</mtu> <txqlen>50</txqlen> </link> <firewall> <zone>secure</zone> </firewall> <ipv4:dhcp> <enabled>true</enabled> <acquire-timeout>15</acquire-timeout> </ipv4:dhcp> </interface> As you can see, options specific to the physical device are grouped in one XML element, generic link-layer options in another, and firewall settings in yet another element. *** Firewall configuration often has nothing to do with particular interface. It also can be arbitrarily complex. Dumbing it down to per-interface setting is extremely limiting.

On 2012-08-29 6:17, Denys Vlasenko wrote: <snip> ... There appears to be some cognitive dissonance in the document. It seems odd to criticize NetworkManager solely for being a bloated, complex daemon, and then, having dismissed NM, in describing the ideal next-generation network management framework, declare that it ought to be a bloated complex daemon... -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net

On Thu, 2012-08-30 at 07:44 +0200, Olaf Kirch wrote: That's NM's heritage, but that's certainly not its focus. We're actually focusing a *lot* more on enterprise use-cases these days, including lots of knobs that admins like to turn. The main coexistence problems center around two things: DNS and the default route. In *any* system you need one thing to manage these, because they are a shared resource. Which interface gets default route priority? Which interface's DNS servers get priority? If the thing that manages these doesn't have the full information available to it, like the user preferences for these options, then you'll end up not making the right choices. The thing that manages these must have information about *all* interfaces. *That's* the problem with multiple competing network management systems. If it weren't for this, they'd all be just fine managing the individual interfaces they were told to manage. But unfortunately the default route and DNS are somewhat crucial to networking... NM has historically attempted to manage both of these, because it had a lot of information. And that creates some problems. We're working on ways to mitigate this issue, but it's not a problem that's easily solved when some people like to use one system, and other people like to use another system. And adding another layer of abstraction to the system to mitigate between network management systems doesn't help a lot either.... :( Dan

On Fri, 2012-09-07 at 14:49 -0500, Bruno Wolff III wrote: 'man NetworkManager' is a good place to start here, which leads you to NetworkManager.conf, which leads you to: dns=dnsmasq which with 0.9.6 and any options you put into custom configuration in /etc/NetworkManager/dnsmasq.d, should do exactly what you want. The local caching nameserver functionality has actually been around for quite a few years. Also, on per-connection basis, NM allows appending custom search domains to the list returned by the DHCP server, or as you've found you can hijack the dhclient config files to do what you want too. As always, more documentation would be awesome; we've got a wiki but we can also add stuff to a FAQ in git too. Dan

On Fri, 2012-09-07 at 16:44 -0500, Michael Cronenworth wrote: Honestly, no. And here's why. First, we should probably be using a caching dns server by default. But really, this isn't an option that should ever be in a UI, because it should Just Work and you shouldn't have to care about DNS. Just like we should be auto-detecting proxy servers if we can, instead of making you enter that all manually. If you know you want a local caching DNS server, then you'll probably know how to turn it on. If you don't know what one is, then we should be setting up the right defaults for you. A distro that caters towards "easy to use" should probably just drop a default NM config file with dns=dnsmasq into /etc, while ones that cater more towards sysadmins would not. Dan

On Fri, 2012-09-07 at 16:46 -0500, Bruno Wolff III wrote: In that case, how about we add a plugin for dnscache too? Though we are in the process of figuring out a more generic interface, so that for example dnscache itself could distribute a plugin for NM instead of having to build this stuff into NM. Dan

On Wed, 2012-08-29 at 15:17 +0200, Denys Vlasenko wrote: Historically it didn't require daemons, but these days networking is a lot more complex, and people often run multiple interfaces, and something gets to mediate between multiple interfaces. I often argue that the *old* system of a ton of shellscripts and many different processes (dhcp, ppp, vpn, wpa_supplicant, ipsec, etc) all doing their own thing and being loosely tied together via a bunch of cobbled-together shellscripts is *more* complex, because: 1) error handling and reporting is very hard to get right in shell 2) there's about 10x more cracks for things to fall through and problems to occur in a loosely tied together system like this 3) each tool has it's own configuration format and it's own command/control format, and none of these are anything like the others 4) handling events from these tools often requires a daemon-type process anyway 5) half or more of the effort goes into making all these tools work together and talk reliably and to provide a consistent interface on top of all of them, instead of actually making the networking system better There's no reason *why* a system like NetworkManager should be as complex as you think it is, and I'd argue that it's not. Certianly not 10x as complex as the old system. I'm not acutally sure what a "more elegant" setup that you envision would be; any chance you can share some thoughts? Dan

On Wed, 2012-08-29 at 16:48 +0100, Steven Whitehouse wrote: Obviously we can add a couple of abstraction layers to avoid making hard choices and continue to funnel effort into multiple competing projects. That provides choice, but also increases complexity and the number of gaps that things fall through and that problems are caused by. I think mainly becuase it's orthogonal to the problem. NM already doesn't touch routes on interfaces it doesn't manage The real problem is the *default* route, and that's a shared resource. Tagging doesn't help here because only one thing (whatever that is) really should be handling the default route, precisely because it's a shared resource. It wouldn't be nice to create a higher priority default route than the one created by tool A, just like it wouldn't be nice for tool A to create a higher priority default route than that created by tool B. How do you mediate between them? We attempted to make NetworkManager that thing, but obviously that's incomplete since it doesn't have knowledge of all the interface on the system in some cases. But the thing that handles the default route and DNS priority *does* need to have a pretty good knowledge of all interfaces on the system, and their relative priority. Dan

Hi Miloslav, On Wednesday 29 August 2012 18:42:44 Miloslav Trmač wrote: Hm, I beg to differ here. NetworkManager aims at a relatively compact DBus API that hides most of the complexities of the network stack. That, I believe, is one of the major reasons why it hasn't caught on much in an Enterprise world - there's always one little extra knob an admin wants to twiddle. By contrast, I think a good network management framework needs to have a low entry barrier (in terms of writing configuration files) but still expose as many of the tunables and knobs as possible. Sure, I'm not talking about the syntax. As I said, I couldn't care less if its XML or anything else. My point was that the shell variable syntax is unable to express anything but the most basic concepts, and hence our network management remains simplistic. Well, if you take each of these niche things by themselves, they don't count for much. But they add up, and Cloud/Virtualization is really adding a *lot* of complexity. netcf being a fine example of the hoops you need to jump through in order to support the brave new world of virtualization hosts. My point is that the current system has reach a breaking point. We've piled brick on brick, and the stack is teetering dangerously :) And regarding your comment as to adding new features - that is definitely in scope for wicked. It does already allow to change the set of bonding slaves on the fly, and 802.1x support is on the TODO list. Olaf -- Neo didn't bring down the Matrix. SOA did. (soafacts.com) -------------------------------------------- Olaf Kirch - Director SUSE Linux Enterprise; R&D (okir(a)suse.com) SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)

On Wed, 2012-08-29 at 13:58 +0200, Olaf Kirch wrote: Olaf, I'm very interested to learn more about wicked. Can you perhaps itemize the set of features available to wicked (current as well as in-development) that aren't available (or maybe "accessible) in Network Manager? I'm involved in a project that's aiming to improve central management of networking and having a comprehensive network stack wrapped in a D-BUS API sounds almost too good to be true from that perspective. A CIM wrapper around such an API could go a long way in that direction. Do you have a D-BUS API specification available somewhere in a readable form? That would be very helpful.

On Mon, 2012-09-03 at 09:20 +0200, Olaf Kirch wrote: Can you describe in what way it exceeds? Seems comparable to NM here. Interesting; something we're also looking at (at least at lower levels) for NetworkManager, but it also makes it hard to implement a connection-oriented architecture. NM has had decoupled IPv6 and IPv4 for quite a while now too, actually. dan

On Wednesday 29 August 2012 21:56:45 Jóhann B. Guðmundsson wrote: Indeed, and that's where I'd like to go. Which is one of the reasons for choosing dbus as the transport. Olaf -- Neo didn't bring down the Matrix. SOA did. (soafacts.com) -------------------------------------------- Olaf Kirch - Director SUSE Linux Enterprise; R&D (okir(a)suse.com) SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)

Bill Nottingham (notting(a)redhat.com) said: To be clear, I'm not really convinced yet that this is something we need... there is a lot of legacy admin overhead and infrastructure that is highly resistant to change here. Bill

On Thu, 2012-08-30 at 13:23 +0200, Dennis Jacobfeuerborn wrote: We're working on that in the NM space, with management modes in between "unmanaged" and "fully managed". For whatever reason, we'll never be in a position where one or the other solution "wins", so we'll have to figure out ways to coexist. And obviously the goal is to get better than the current situation. Dan

"Jóhann B. Guðmundsson" (johannbg(a)gmail.com) said: There are 15+ years of legacy implementations of the old configuration format and tools expectations. It has been wired into existing documentation, and multiple tools in the existing OS. This says nothing of administrator's kickstart scripts, deployment tools, and existing brains, all of which are highly resistant to change that merely impacts them in the form of swapping config format A for config format B. (Seriously, as Miloslav said - the configuration format is the least interesting part of any change here.) I'm not saying you *can't* do it, but it's also not something to do lightly. Bill

On Mon, 2012-09-03 at 09:20 +0200, Olaf Kirch wrote: Touching an interface depends on your use-cases. For many cases it should be automatically managed without any intervention from the user, and in many other cases, that should be under the control of the administrator. It's definitely not an either/or choice. Yes; we've modified the ifup/ifdown scripts to handle interaction with NetworkManager. Yes; obviously anything that handles network management on Fedora needs to do this. Dan

On Mon, 2012-09-03 at 09:20 +0200, Olaf Kirch wrote: The problem with integrating network management into systemd in some form is that system only wants to handle the "simple" use-cases. Mainly because it's intended to be a lightweight part of the boot process. But when you start handling any "simple" networking, you'll eventually start handling more complex networking too, because users will request it. And then at some point, you're duplicating all the code that a more complex network management daemon is already doing, and that makes things more confusing for everyone. Is this interface managed by systemd? Or is this interface managed by NM/something-else? Or is this interface manually managed? Should you even have to make this choice? What mediates the routing table and the DNS entries if you have systemd managing some interfaces, NM some other interfaces, and manually managed interfaces too? Another daemon? Yay! Piles of daemons! And since this is Linux, everyone wants to use each thing in isolation from the others, so you can't depend on anything else being on the system. Talking about low-level networking in systemd or whatever flavor-of-the-year initsystem there happens to be necessarily brings up questions like this. I'm not sure the right place for this sort of thing is in the init system. Instead, we should be creating a lightweight-enough network management daemon that could be included on these resource-constrained systems along with systemd to do that. Which is also something we're working towards with NetworkManager. It's actually been possible for a long time to build/run NM without any of [wpa-supplicant, ppp, ConsoleKit, PolicyKit, bluez, etc] if you want a stripped-down system. Dan

On Fri, 2012-09-07 at 21:02 +0000, "Jóhann B. Guðmundsson" wrote: I don't think the init system should really be responsible for a ton of the network management, otherwise you'll have people like Denys V. saying it's too bloated and should be simpler. I'm not really sure that moving everything a network management system *should* be doing into the init system solves any problems WRT complexity. There's clearly a gray area here, but my assertion is that if you put a significant amount of logic into the init system, where does it end? If you don't add bridging, then people are going to request bridging in the init system. And if you add it there, then why not add bonding? Where does it end? Next, for the things it *doesn't* do, how does it interact with the other management system that *does* do those things? What mediates the shared resources like DNS and the default route? And they all have to use the same configuration formats too. Because everyone is different, everyone has their own requirements and a lot of people don't care about anything *except* their own requirements. Saying "you have to take an extra step after plugging your network device in because server-type people need it to work this way" is just as much a non-starter as saying "you have to upgrade your kernel driver because we made this option automatic instead of manual so desktop users don't have to toggle a checkbox" to server people. That's Free Software. If we all agreed, 6 months later somebody would come up with a different system because they didn't like D-Bus, didnt' like the base library we were using (glib/qt/whatever), didn't like the API or the configuration model, etc. There's room to work towards consensus on functionality, but at the moment I don't see a lot of momentum towards a Grand New Unified Plan, partially because that's how free software works. It would be great if we had one, and I'm certainly willing to entertain changes to NetworkManager that make it more capable for anyone's use-cases. Nobody thinks NM has reached perfection, least of all me. Dan

On 09/09/2012 07:14 AM, Nathanael D. Noblet wrote: +1 (same experience here), big kudos to Dan. -- Jiri