Hi all, The kubevirt networking team is working on a new cmd/lib tool to generate dynamic NMState configurations. When networking configuration for a cluster is needed and all the details are common between the nodes in the cluster a NMState yaml configuration is enough. Problems arise when some of the network configuration details are different between nodes and depend on the current node network state. For that a different NMState yaml configuration needs to be generated per node and that's not convenient for big clusters and also at scale up scenarios. The NMPolicy goal is to solve this problem. Given a node network state and a network configuration policy (common to the cluster), the NMPolicy tool will generate a node specific desired network state. NMPolicy was already integrated to kubernetes-nmstate and a usage example can be found here <https://github.com/nmstate/kubernetes-nmstate/blob/main/test/e2e/handler/... . *CLI* Usage example - Simulated --help output ./nmpolicyctl -h This tool helps you to generate dynamic NMState configurations. The generated NMState configuration is written to *STDOUT*. Usage of ./*nmpolicyctl*: Commands: *gen* Generates NMState by policy filename Flags: -s, --current-state=current-state.yaml: input file path to current NMState. If not specified, stdin is used. -c, --captured-state-output=captured-state-output.yaml: output file path to the emitted captured states. If not specified, ~/.cache/nmpolicy/cache.yaml will be used. *Creating a new bridge with the base interface as its port. The bridge will get the mac address of the base interface.* # echo """ capture: *default-gw: routes.running.destination=="0.0.0.0/0 <http://0.0.0.0/0>"* * base-iface: interfaces.name <http://interfaces.name>==capture.default-gw.routes.running.0.next-hop-... desiredState: interfaces: - name: br1 description: Linux bridge with base interface as a port type: linux-bridge state: up mac-address: "{{ *capture.base-iface.interfaces.**0.mac-address* }}" ipv4: dhcp: true enabled: true bridge: options: stp: enabled: false port: - name: "{{ *capture.base-iface.interfaces.0.name <http://capture.base-iface.interfaces.0.name>* }}" """ > *policy.yaml* *# nmstatectl show | nmpolicyctl gen policy.yaml | nmstatectl apply* *Output of - nmstatectl show | nmpolicyctl gen policy.yaml*

base-iface: metaInfo: time: "2021-12-15T13:45:40Z" state: interfaces: - accept-all-mac-addresses: false ethernet: auto-negotiation: false ethtool: feature: rx-gro: true rx-gro-list: false rx-udp-gro-forwarding: false tx-checksum-ip-generic: true tx-generic-segmentation: true tx-nocache-copy: false tx-tcp-ecn-segmentation: true tx-tcp-mangleid-segmentation: false tx-tcp-segmentation: true tx-tcp6-segmentation: true ring: rx: 256 tx: 256 ipv4: address: - ip: 192.168.66.101 prefix-length: 24 auto-dns: true auto-gateway: true auto-route-table-id: 0 auto-routes: true dhcp: true enabled: true ipv6: address: - ip: fd00::101 prefix-length: 128 - ip: fe80::2d55:7c37:8090:7261 prefix-length: 64 auto-dns: true auto-gateway: true auto-route-table-id: 0 auto-routes: true autoconf: true dhcp: true enabled: true lldp: enabled: false *mac-address: 52:55:00:D1:55:01* mtu: 1500 *name: eth0* state: up type: ethernet default-gw: metaInfo: time: "2021-12-15T13:45:40Z" state: routes: running: - *destination: 0.0.0.0/0 <http://0.0.0.0/0>* metric: 100 next-hop-address: 192.168.66.2 *next-hop-interface: eth0* table-id: 254 *Stdout* (the *desired state* to be applied by nmstatectl) interfaces: - bridge: options: stp: enabled: false port: - *name: eth0* vlan: mode: trunk trunk-tags: - id-range: max: 4094 min: 2 description: Linux bridge with base interface as a port ipv4: dhcp: true enabled: true *mac-address: 52:55:00:D1:55:01* *name: br1* state: up type: linux-bridge For more details, please review the design doc <https://docs.google.com/document/d/1k8gljRJoGtCGXoQ3YkTpDmekLAIeUWZ4P3-oP... and share your feedback about it. Thanks, Alona. -- You received this message because you are subscribed to the Google Groups "kubevirt-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev+unsubscribe(a)googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/CAJOMPacxNke2pJnjJ1PKxz0f%... <https://groups.google.com/d/msgid/kubevirt-dev/CAJOMPacxNke2pJnjJ1PKxz0f%... .

On Fri, Dec 17, 2021 at 3:58 PM Dan Kenigsberg <danken(a)redhat.com&gt; wrote: > Thanks for sharing your plan. > > I would like to create a Linux bond of all the NICs of my machine, create > a Linux bridge on top of it, and copy my original mac/IP/routing to that > bridge. Can you share a policy that would let me express this desired state? > You need to know the number of NICs, for example for three of them this would be it.

apiVersion: nmstate.io/v1 kind: NodeNetworkConfigurationPolicy metadata: name: bra1 spec: capture: default-gw: routes.running.destination=="0.0.0.0/0" primary-iface: interfaces.name ==capture.default-gw.routes.running.0.next-hop-interface ethernet-ifaces: interface.type==ethernet desiredState: interfaces: - name: bond1 type: bond state: up link-aggregation: mode: balance-rr options: miimon: '140' port: - name: "{{ capture.ethernet-ifaces.interfaces.0.name }}" - name: "{{ capture.ethernet-ifaces.interfaces.1.name }}" - name: "{{ capture.ethernet-ifaces.interfaces.2.name }}" - name: br1 description: Linux bridge with base interface as a port type: linux-bridge state: up mac-address: "{{ capture.primary-iface.interfaces.0.mac-address }}" ipv4: "{{ capture.primary-iface.interfaces.0.ipv4 }}" bridge: options: stp: enabled: false port: - bondall

> > Regards, > Dan. > > On Thu, Dec 16, 2021 at 11:24 AM Alona Paz <alkaplan(a)redhat.com&gt; wrote: > >> >> Hi all, >> >> The kubevirt networking team is working on a new cmd/lib tool to >> generate dynamic NMState configurations. >> >> When networking configuration for a cluster is needed and all the >> details are common between the nodes in the cluster a NMState yaml >> configuration is enough. >> >> Problems arise when some of the network configuration details are >> different between nodes and depend on the current node network state. >> >> For that a different NMState yaml configuration needs to be generated >> per node and that's not convenient for big clusters and also at scale up >> scenarios. >> >> >> The NMPolicy goal is to solve this problem. >> >> Given a node network state and a network configuration policy (common >> to the cluster), the NMPolicy tool will generate a node specific desired >> network state. >> >> >> NMPolicy was already integrated to kubernetes-nmstate and a usage >> example can be found here >> <https://github.com/nmstate/kubernetes-nmstate/blob/main/test/e2e/handler/... >> . >> >> >> *CLI* Usage example - >> >> >> Simulated --help output >> >> ./nmpolicyctl -h >> >> This tool helps you to generate dynamic NMState configurations. >> >> The generated NMState configuration is written to *STDOUT*. >> >> Usage of ./*nmpolicyctl*: >> >> Commands: >> >> *gen* Generates NMState by policy filename >> >> Flags: >> >> -s, --current-state=current-state.yaml: input file path to current >> NMState. If not specified, stdin is used. >> >> -c, --captured-state-output=captured-state-output.yaml: output file >> path to the emitted captured states. If not specified, >> ~/.cache/nmpolicy/cache.yaml will be used. >> >> *Creating a new bridge with the base interface as its port. The bridge >> will get the mac address of the base interface.* >> >> # echo """ >> >> capture: >> >> *default-gw: routes.running.destination=="0.0.0.0/0 >> <http://0.0.0.0/0>"* >> >> * base-iface: interfaces.name >> <http://interfaces.name>==capture.default-gw.routes.running.0.next-hop-... >> >> desiredState: >> >> interfaces: >> >> - name: br1 >> >> description: Linux bridge with base interface as a port >> >> type: linux-bridge >> >> state: up >> >> mac-address: "{{ *capture.base-iface.interfaces.**0.mac-address* }}" >> >> ipv4: >> >> dhcp: true >> >> enabled: true >> >> bridge: >> >> options: >> >> stp: >> >> enabled: false >> >> port: >> >> - name: "{{ *capture.base-iface.interfaces.0.name >> <http://capture.base-iface.interfaces.0.name>* }}" >> >> """ > *policy.yaml* >> >> *# nmstatectl show | nmpolicyctl gen policy.yaml | nmstatectl apply* >> >> >> *Output of - nmstatectl show | nmpolicyctl gen policy.yaml* >> >> >> > ~/.cache/nmpolicy/*cache*.yaml (the default captured states output file) >> >> base-iface: >> >> metaInfo: >> >> time: "2021-12-15T13:45:40Z" >> >> state: >> >> interfaces: >> >> - accept-all-mac-addresses: false >> >> ethernet: >> >> auto-negotiation: false >> >> ethtool: >> >> feature: >> >> rx-gro: true >> >> rx-gro-list: false >> >> rx-udp-gro-forwarding: false >> >> tx-checksum-ip-generic: true >> >> tx-generic-segmentation: true >> >> tx-nocache-copy: false >> >> tx-tcp-ecn-segmentation: true >> >> tx-tcp-mangleid-segmentation: false >> >> tx-tcp-segmentation: true >> >> tx-tcp6-segmentation: true >> >> ring: >> >> rx: 256 >> >> tx: 256 >> >> ipv4: >> >> address: >> >> - ip: 192.168.66.101 >> >> prefix-length: 24 >> >> auto-dns: true >> >> auto-gateway: true >> >> auto-route-table-id: 0 >> >> auto-routes: true >> >> dhcp: true >> >> enabled: true >> >> ipv6: >> >> address: >> >> - ip: fd00::101 >> >> prefix-length: 128 >> >> - ip: fe80::2d55:7c37:8090:7261 >> >> prefix-length: 64 >> >> auto-dns: true >> >> auto-gateway: true >> >> auto-route-table-id: 0 >> >> auto-routes: true >> >> autoconf: true >> >> dhcp: true >> >> enabled: true >> >> lldp: >> >> enabled: false >> >> *mac-address: 52:55:00:D1:55:01* >> >> mtu: 1500 >> >> *name: eth0* >> >> state: up >> >> type: ethernet >> >> default-gw: >> >> metaInfo: >> >> time: "2021-12-15T13:45:40Z" >> >> state: >> >> routes: >> >> running: >> >> - *destination: 0.0.0.0/0 <http://0.0.0.0/0>* >> >> metric: 100 >> >> next-hop-address: 192.168.66.2 >> >> *next-hop-interface: eth0* >> >> table-id: 254 >> >> >> *Stdout* (the *desired state* to be applied by nmstatectl) >> >> interfaces: >> >> - bridge: >> >> options: >> >> stp: >> >> enabled: false >> >> port: >> >> - *name: eth0* >> >> vlan: >> >> mode: trunk >> >> trunk-tags: >> >> - id-range: >> >> max: 4094 >> >> min: 2 >> >> description: Linux bridge with base interface as a port >> >> ipv4: >> >> dhcp: true >> >> enabled: true >> >> *mac-address: 52:55:00:D1:55:01* >> >> *name: br1* >> >> state: up >> >> type: linux-bridge >> >> >> >> For more details, please review the design doc >> <https://docs.google.com/document/d/1k8gljRJoGtCGXoQ3YkTpDmekLAIeUWZ4P3-oP... >> and share your feedback about it. >> >> Thanks, >> Alona. >> >> -- >> You received this message because you are subscribed to the Google >> Groups "kubevirt-dev" group. >> To unsubscribe from this group and stop receiving emails from it, send >> an email to kubevirt-dev+unsubscribe(a)googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/kubevirt-dev/CAJOMPacxNke2pJnjJ1PKxz0f%... >> <https://groups.google.com/d/msgid/kubevirt-dev/CAJOMPacxNke2pJnjJ1PKxz0f%... >> . >> > -- > You received this message because you are subscribed to the Google Groups > "kubevirt-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to kubevirt-dev+unsubscribe(a)googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/kubevirt-dev/CAHOEP56e4zhHPtP8rZp%3D%3D... > <https://groups.google.com/d/msgid/kubevirt-dev/CAHOEP56e4zhHPtP8rZp%3D%3D... > . > -- *Quique Llorente* CNV networking Senior Software Engineer Red Hat EMEA <https://www.redhat.com/> ellorent(a)redhat.com <arxcruz(a)redhat.com&gt; @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com/>

On Sun, Dec 19, 2021 at 10:42 AM Dan Kenigsberg <danken(a)redhat.com&gt; wrote: > > > On Fri, Dec 17, 2021 at 5:25 PM Felix Enrique Llorente Pastora < > ellorent(a)redhat.com&gt; wrote: > >> >> >> On Fri, Dec 17, 2021 at 3:58 PM Dan Kenigsberg <danken(a)redhat.com&gt; >> wrote: >> >>> Thanks for sharing your plan. >>> >>> I would like to create a Linux bond of all the NICs of my machine, >>> create a Linux bridge on top of it, and copy my original mac/IP/routing to >>> that bridge. Can you share a policy that would let me express this desired >>> state? >>> >> >> You need to know the number of NICs, for example for three of them this >> would be it. >> > > Thanks. Having a Python-like `0:-1` slice index to select a sublist could > be even cooler. >

> More importantly, I would like to be able to select all interfaces which > have vlan 204 enabled on their switch port, as reported by lldp. What would > the capture look like in this case? > capture: vlan-204: interfaces.lldp.enabled=="true" | interfaces.lldp.neighbors.ieee-802-1-vlans.vid==204 But I suspect our current implementation will fail resolving this capture since not all the neighbor entries have the "ieee-802-1-vlans" key. We can change the implementation not to fail if at least one of the entries in a list has the required path. Then, even the following capture should work- capture: vlan-204: interfaces.lldp.neighbors.ieee-802-1-vlans.vid==204 > Is the suggested language expressive enough to capture them? Here's my > (redacted) nmstate show for one such interface. I find the LLDP portion > there a bit confusing. > > - ethernet: > auto-negotiation: true > duplex: full > speed: 10000 > ipv4: > address: [] > auto-dns: true > auto-gateway: true > auto-route-table-id: 0 > auto-routes: true > dhcp: true > enabled: true > ipv6: > address: [] > autoconf: false > dhcp: false > enabled: false > lldp: > enabled: true > neighbors: > - - system-name: sw01-access-f1.example.com > type: 5 > - system-description: 'Juniper Networks, Inc' > type: 6 > - system-capabilities: > - MAC Bridge component > - Router > type: 7 > - _description: MAC address > chassis-id-type: 4 > type: 1 > - _description: Interface name > port-id: xe-0/0/17 > port-id-type: 5 > type: 2 > - ieee-802-1-vlans: > - name: vlan-204 > vid: 204 > - name: vlan-205 > vid: 205 > subtype: 3 > type: 127 > - ieee-802-3-mac-phy-conf: > autoneg: false > operational-mau-type: 0 > pmd-autoneg-cap: 32768 > oui: 00:12:0f > subtype: 1 > type: 127 > - management-addresses: > address-subtype: ipv4 > interface-number: 0 > interface-number-subtype: 1 > type: 8 > - ieee-802-3-max-frame-size: 9216 > oui: 00:12:0f > subtype: 4 > type: 127 > mtu: 1500 > name: ens1f0 > state: up > type: ethernet > > >> apiVersion: nmstate.io/v1 >> >> kind: NodeNetworkConfigurationPolicy >> >> metadata: >> >> name: bra1 >> >> spec: >> >> capture: >> >> default-gw: routes.running.destination=="0.0.0.0/0" >> >> primary-iface: interfaces.name >> ==capture.default-gw.routes.running.0.next-hop-interface >> ethernet-ifaces: interface.type==ethernet >> >> desiredState: >> >> interfaces: >> >> - name: bond1 >> >> type: bond >> >> state: up >> >> link-aggregation: >> >> mode: balance-rr >> >> options: >> >> miimon: '140' >> >> port: >> >> - name: "{{ capture.ethernet-ifaces.interfaces.0.name }}" >> >> - name: "{{ capture.ethernet-ifaces.interfaces.1.name }}" >> >> - name: "{{ capture.ethernet-ifaces.interfaces.2.name }}" >> >> - name: br1 >> >> description: Linux bridge with base interface as a port >> >> type: linux-bridge >> >> state: up >> >> mac-address: "{{ capture.primary-iface.interfaces.0.mac-address >> }}" >> ipv4: "{{ capture.primary-iface.interfaces.0.ipv4 }}" >> >> bridge: >> >> options: >> >> stp: >> >> enabled: false >> >> port: >> >> - bondall >> > > I suspect you need `bond1` here? > yes > >> >> >> >> >>> >>> Regards, >>> Dan. >>> >>> On Thu, Dec 16, 2021 at 11:24 AM Alona Paz <alkaplan(a)redhat.com&gt; wrote: >>> >>>> >>>> Hi all, >>>> >>>> The kubevirt networking team is working on a new cmd/lib tool to >>>> generate dynamic NMState configurations. >>>> >>>> When networking configuration for a cluster is needed and all the >>>> details are common between the nodes in the cluster a NMState yaml >>>> configuration is enough. >>>> >>>> Problems arise when some of the network configuration details are >>>> different between nodes and depend on the current node network state. >>>> >>>> For that a different NMState yaml configuration needs to be generated >>>> per node and that's not convenient for big clusters and also at scale up >>>> scenarios. >>>> >>>> >>>> The NMPolicy goal is to solve this problem. >>>> >>>> Given a node network state and a network configuration policy (common >>>> to the cluster), the NMPolicy tool will generate a node specific desired >>>> network state. >>>> >>>> >>>> NMPolicy was already integrated to kubernetes-nmstate and a usage >>>> example can be found here >>>> <https://github.com/nmstate/kubernetes-nmstate/blob/main/test/e2e/handler/... >>>> . >>>> >>>> >>>> *CLI* Usage example - >>>> >>>> >>>> Simulated --help output >>>> >>>> ./nmpolicyctl -h >>>> >>>> This tool helps you to generate dynamic NMState configurations. >>>> >>>> The generated NMState configuration is written to *STDOUT*. >>>> >>>> Usage of ./*nmpolicyctl*: >>>> >>>> Commands: >>>> >>>> *gen* Generates NMState by policy filename >>>> >>>> Flags: >>>> >>>> -s, --current-state=current-state.yaml: input file path to current >>>> NMState. If not specified, stdin is used. >>>> >>>> -c, --captured-state-output=captured-state-output.yaml: output >>>> file path to the emitted captured states. If not specified, >>>> ~/.cache/nmpolicy/cache.yaml will be used. >>>> >>>> *Creating a new bridge with the base interface as its port. The bridge >>>> will get the mac address of the base interface.* >>>> >>>> # echo """ >>>> >>>> capture: >>>> >>>> *default-gw: routes.running.destination=="0.0.0.0/0 >>>> <http://0.0.0.0/0>"* >>>> >>>> * base-iface: interfaces.name >>>> <http://interfaces.name>==capture.default-gw.routes.running.0.next-hop-... >>>> >>>> desiredState: >>>> >>>> interfaces: >>>> >>>> - name: br1 >>>> >>>> description: Linux bridge with base interface as a port >>>> >>>> type: linux-bridge >>>> >>>> state: up >>>> >>>> mac-address: "{{ *capture.base-iface.interfaces.**0.mac-address* >>>> }}" >>>> >>>> ipv4: >>>> >>>> dhcp: true >>>> >>>> enabled: true >>>> >>>> bridge: >>>> >>>> options: >>>> >>>> stp: >>>> >>>> enabled: false >>>> >>>> port: >>>> >>>> - name: "{{ *capture.base-iface.interfaces.0.name >>>> <http://capture.base-iface.interfaces.0.name>* }}" >>>> >>>> """ > *policy.yaml* >>>> >>>> *# nmstatectl show | nmpolicyctl gen policy.yaml | nmstatectl apply* >>>> >>>> >>>> *Output of - nmstatectl show | nmpolicyctl gen policy.yaml* >>>> >>>> >>>> >>> ~/.cache/nmpolicy/*cache*.yaml (the default captured states output >>>> file) >>>> >>>> base-iface: >>>> >>>> metaInfo: >>>> >>>> time: "2021-12-15T13:45:40Z" >>>> >>>> state: >>>> >>>> interfaces: >>>> >>>> - accept-all-mac-addresses: false >>>> >>>> ethernet: >>>> >>>> auto-negotiation: false >>>> >>>> ethtool: >>>> >>>> feature: >>>> >>>> rx-gro: true >>>> >>>> rx-gro-list: false >>>> >>>> rx-udp-gro-forwarding: false >>>> >>>> tx-checksum-ip-generic: true >>>> >>>> tx-generic-segmentation: true >>>> >>>> tx-nocache-copy: false >>>> >>>> tx-tcp-ecn-segmentation: true >>>> >>>> tx-tcp-mangleid-segmentation: false >>>> >>>> tx-tcp-segmentation: true >>>> >>>> tx-tcp6-segmentation: true >>>> >>>> ring: >>>> >>>> rx: 256 >>>> >>>> tx: 256 >>>> >>>> ipv4: >>>> >>>> address: >>>> >>>> - ip: 192.168.66.101 >>>> >>>> prefix-length: 24 >>>> >>>> auto-dns: true >>>> >>>> auto-gateway: true >>>> >>>> auto-route-table-id: 0 >>>> >>>> auto-routes: true >>>> >>>> dhcp: true >>>> >>>> enabled: true >>>> >>>> ipv6: >>>> >>>> address: >>>> >>>> - ip: fd00::101 >>>> >>>> prefix-length: 128 >>>> >>>> - ip: fe80::2d55:7c37:8090:7261 >>>> >>>> prefix-length: 64 >>>> >>>> auto-dns: true >>>> >>>> auto-gateway: true >>>> >>>> auto-route-table-id: 0 >>>> >>>> auto-routes: true >>>> >>>> autoconf: true >>>> >>>> dhcp: true >>>> >>>> enabled: true >>>> >>>> lldp: >>>> >>>> enabled: false >>>> >>>> *mac-address: 52:55:00:D1:55:01* >>>> >>>> mtu: 1500 >>>> >>>> *name: eth0* >>>> >>>> state: up >>>> >>>> type: ethernet >>>> >>>> default-gw: >>>> >>>> metaInfo: >>>> >>>> time: "2021-12-15T13:45:40Z" >>>> >>>> state: >>>> >>>> routes: >>>> >>>> running: >>>> >>>> - *destination: 0.0.0.0/0 <http://0.0.0.0/0>* >>>> >>>> metric: 100 >>>> >>>> next-hop-address: 192.168.66.2 >>>> >>>> *next-hop-interface: eth0* >>>> >>>> table-id: 254 >>>> >>>> >>>> *Stdout* (the *desired state* to be applied by nmstatectl) >>>> >>>> interfaces: >>>> >>>> - bridge: >>>> >>>> options: >>>> >>>> stp: >>>> >>>> enabled: false >>>> >>>> port: >>>> >>>> - *name: eth0* >>>> >>>> vlan: >>>> >>>> mode: trunk >>>> >>>> trunk-tags: >>>> >>>> - id-range: >>>> >>>> max: 4094 >>>> >>>> min: 2 >>>> >>>> description: Linux bridge with base interface as a port >>>> >>>> ipv4: >>>> >>>> dhcp: true >>>> >>>> enabled: true >>>> >>>> *mac-address: 52:55:00:D1:55:01* >>>> >>>> *name: br1* >>>> >>>> state: up >>>> >>>> type: linux-bridge >>>> >>>> >>>> >>>> For more details, please review the design doc >>>> <https://docs.google.com/document/d/1k8gljRJoGtCGXoQ3YkTpDmekLAIeUWZ4P3-oP... >>>> and share your feedback about it. >>>> >>>> Thanks, >>>> Alona. >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "kubevirt-dev" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to kubevirt-dev+unsubscribe(a)googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/kubevirt-dev/CAJOMPacxNke2pJnjJ1PKxz0f%... >>>> <https://groups.google.com/d/msgid/kubevirt-dev/CAJOMPacxNke2pJnjJ1PKxz0f%... >>>> . >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "kubevirt-dev" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to kubevirt-dev+unsubscribe(a)googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/kubevirt-dev/CAHOEP56e4zhHPtP8rZp%3D%3D... >>> <https://groups.google.com/d/msgid/kubevirt-dev/CAHOEP56e4zhHPtP8rZp%3D%3D... >>> . >>> >> >> >> -- >> *Quique Llorente* >> >> CNV networking Senior Software Engineer >> >> Red Hat EMEA <https://www.redhat.com/> >> >> ellorent(a)redhat.com <arxcruz(a)redhat.com&gt; >> @RedHat <https://twitter.com/redhat> Red Hat >> <https://www.linkedin.com/company/red-hat> Red Hat >> <https://www.facebook.com/RedHatInc> >> <https://www.redhat.com/> >> >

On Sun, Dec 19, 2021 at 11:20 AM Alona Paz <alkaplan(a)redhat.com&gt; wrote: > > > > On Sun, Dec 19, 2021 at 10:42 AM Dan Kenigsberg <danken(a)redhat.com&gt; wrote: >> >> >> >> On Fri, Dec 17, 2021 at 5:25 PM Felix Enrique Llorente Pastora < ellorent(a)redhat.com&gt; wrote: >>> >>> >>> >>> On Fri, Dec 17, 2021 at 3:58 PM Dan Kenigsberg <danken(a)redhat.com&gt; wrote: >>>> >>>> Thanks for sharing your plan. >>>> >>>> I would like to create a Linux bond of all the NICs of my machine, create a Linux bridge on top of it, and copy my original mac/IP/routing to that bridge. Can you share a policy that would let me express this desired state? >>> >>> >>> You need to know the number of NICs, for example for three of them this would be it. >> >> >> Thanks. Having a Python-like `0:-1` slice index to select a sublist could be even cooler. >> >> More importantly, I would like to be able to select all interfaces which have vlan 204 enabled on their switch port, as reported by lldp. What would the capture look like in this case? > > > capture: > vlan-204: interfaces.lldp.enabled=="true" | interfaces.lldp.neighbors.ieee-802-1-vlans.vid==204 > > But I suspect our current implementation will fail resolving this capture since not all the neighbor entries have the "ieee-802-1-vlans" key. > We can change the implementation not to fail if at least one of the entries in a list has the required path. > > Then, even the following capture should work- > > capture: > vlan-204: interfaces.lldp.neighbors.ieee-802-1-vlans.vid==204 Thanks. I find the capture language very powerful, I think I would benefit from a tool that helps me build them, based on the current state. E.g a command line such as nmstatectl show | nmpolicy testcapture 'interfaces.lldp.enabled=="true" | interfaces.lldp.neighbors.ieee-802-1-vlans.vid==204' that would emit the value of yaml-encoded captures. Would something like this be possible?

> > >> >> Is the suggested language expressive enough to capture them? Here's my (redacted) nmstate show for one such interface. I find the LLDP portion there a bit confusing. >> >> - ethernet: >> auto-negotiation: true >> duplex: full >> speed: 10000 >> ipv4: >> address: [] >> auto-dns: true >> auto-gateway: true >> auto-route-table-id: 0 >> auto-routes: true >> dhcp: true >> enabled: true >> ipv6: >> address: [] >> autoconf: false >> dhcp: false >> enabled: false >> lldp: >> enabled: true >> neighbors: >> - - system-name: sw01-access-f1.example.com >> type: 5 >> - system-description: 'Juniper Networks, Inc' >> type: 6 >> - system-capabilities: >> - MAC Bridge component >> - Router >> type: 7 >> - _description: MAC address >> chassis-id-type: 4 >> type: 1 >> - _description: Interface name >> port-id: xe-0/0/17 >> port-id-type: 5 >> type: 2 >> - ieee-802-1-vlans: >> - name: vlan-204 >> vid: 204 >> - name: vlan-205 >> vid: 205 >> subtype: 3 >> type: 127 >> - ieee-802-3-mac-phy-conf: >> autoneg: false >> operational-mau-type: 0 >> pmd-autoneg-cap: 32768 >> oui: 00:12:0f >> subtype: 1 >> type: 127 >> - management-addresses: >> address-subtype: ipv4 >> interface-number: 0 >> interface-number-subtype: 1 >> type: 8 >> - ieee-802-3-max-frame-size: 9216 >> oui: 00:12:0f >> subtype: 4 >> type: 127 >> mtu: 1500 >> name: ens1f0 >> state: up >> type: ethernet >> >>> >>> apiVersion: nmstate.io/v1 >>> kind: NodeNetworkConfigurationPolicy >>> metadata: >>> name: bra1 >>> spec: >>> capture: >>> default-gw: routes.running.destination=="0.0.0.0/0" >>> primary-iface: interfaces.name ==capture.default-gw.routes.running.0.next-hop-interface >>> ethernet-ifaces: interface.type==ethernet >>> desiredState: >>> interfaces: >>> - name: bond1 >>> type: bond >>> state: up >>> link-aggregation: >>> mode: balance-rr >>> options: >>> miimon: '140' >>> port: >>> - name: "{{ capture.ethernet-ifaces.interfaces.0.name }}" >>> - name: "{{ capture.ethernet-ifaces.interfaces.1.name }}" >>> - name: "{{ capture.ethernet-ifaces.interfaces.2.name }}" >>> - name: br1 >>> description: Linux bridge with base interface as a port >>> type: linux-bridge >>> state: up >>> mac-address: "{{ capture.primary-iface.interfaces.0.mac-address }}" >>> ipv4: "{{ capture.primary-iface.interfaces.0.ipv4 }}" >>> bridge: >>> options: >>> stp: >>> enabled: false >>> port: >>> - bondall >> >> >> I suspect you need `bond1` here? > > > yes > >> >>> >>> >>> >>>> >>>> >>>> Regards, >>>> Dan. >>>> >>>> On Thu, Dec 16, 2021 at 11:24 AM Alona Paz <alkaplan(a)redhat.com&gt; wrote: >>>>> >>>>> >>>>> Hi all, >>>>> >>>>> The kubevirt networking team is working on a new cmd/lib tool to generate dynamic NMState configurations. >>>>> >>>>> When networking configuration for a cluster is needed and all the details are common between the nodes in the cluster a NMState yaml configuration is enough. >>>>> >>>>> Problems arise when some of the network configuration details are different between nodes and depend on the current node network state. >>>>> >>>>> For that a different NMState yaml configuration needs to be generated per node and that's not convenient for big clusters and also at scale up scenarios. >>>>> >>>>> >>>>> The NMPolicy goal is to solve this problem. >>>>> >>>>> Given a node network state and a network configuration policy (common to the cluster), the NMPolicy tool will generate a node specific desired network state. >>>>> >>>>> >>>>> NMPolicy was already integrated to kubernetes-nmstate and a usage example can be found here. >>>>> >>>>> >>>>> CLI Usage example - >>>>> >>>>> >>>>> Simulated --help output >>>>> >>>>> ./nmpolicyctl -h >>>>> >>>>> This tool helps you to generate dynamic NMState configurations. >>>>> >>>>> The generated NMState configuration is written to STDOUT. >>>>> >>>>> >>>>> Usage of ./nmpolicyctl: >>>>> >>>>> Commands: >>>>> >>>>> gen Generates NMState by policy filename >>>>> >>>>> Flags: >>>>> >>>>> -s, --current-state=current-state.yaml: input file path to current NMState. If not specified, stdin is used. >>>>> >>>>> -c, --captured-state-output=captured-state-output.yaml: output file path to the emitted captured states. If not specified, ~/.cache/nmpolicy/cache.yaml will be used. >>>>> >>>>> >>>>> Creating a new bridge with the base interface as its port. The bridge will get the mac address of the base interface. >>>>> >>>>> # echo """ >>>>> >>>>> capture: >>>>> >>>>> default-gw: routes.running.destination=="0.0.0.0/0" >>>>> >>>>> base-iface: interfaces.name ==capture.default-gw.routes.running.0.next-hop-interface >>>>> >>>>> desiredState: >>>>> >>>>> interfaces: >>>>> >>>>> - name: br1 >>>>> >>>>> description: Linux bridge with base interface as a port >>>>> >>>>> type: linux-bridge >>>>> >>>>> state: up >>>>> >>>>> mac-address: "{{ capture.base-iface.interfaces.0.mac-address }}" >>>>> >>>>> ipv4: >>>>> >>>>> dhcp: true >>>>> >>>>> enabled: true >>>>> >>>>> bridge: >>>>> >>>>> options: >>>>> >>>>> stp: >>>>> >>>>> enabled: false >>>>> >>>>> port: >>>>> >>>>> - name: "{{ capture.base-iface.interfaces.0.name }}" >>>>> >>>>> """ > policy.yaml >>>>> >>>>> >>>>> # nmstatectl show | nmpolicyctl gen policy.yaml | nmstatectl apply >>>>> >>>>> >>>>> >>>>> Output of - nmstatectl show | nmpolicyctl gen policy.yaml >>>>> >>>>> >>>>> >>>>> ~/.cache/nmpolicy/cache.yaml (the default captured states output file) >>>>> >>>>> base-iface: >>>>> >>>>> metaInfo: >>>>> >>>>> time: "2021-12-15T13:45:40Z" >>>>> >>>>> state: >>>>> >>>>> interfaces: >>>>> >>>>> - accept-all-mac-addresses: false >>>>> >>>>> ethernet: >>>>> >>>>> auto-negotiation: false >>>>> >>>>> ethtool: >>>>> >>>>> feature: >>>>> >>>>> rx-gro: true >>>>> >>>>> rx-gro-list: false >>>>> >>>>> rx-udp-gro-forwarding: false >>>>> >>>>> tx-checksum-ip-generic: true >>>>> >>>>> tx-generic-segmentation: true >>>>> >>>>> tx-nocache-copy: false >>>>> >>>>> tx-tcp-ecn-segmentation: true >>>>> >>>>> tx-tcp-mangleid-segmentation: false >>>>> >>>>> tx-tcp-segmentation: true >>>>> >>>>> tx-tcp6-segmentation: true >>>>> >>>>> ring: >>>>> >>>>> rx: 256 >>>>> >>>>> tx: 256 >>>>> >>>>> ipv4: >>>>> >>>>> address: >>>>> >>>>> - ip: 192.168.66.101 >>>>> >>>>> prefix-length: 24 >>>>> >>>>> auto-dns: true >>>>> >>>>> auto-gateway: true >>>>> >>>>> auto-route-table-id: 0 >>>>> >>>>> auto-routes: true >>>>> >>>>> dhcp: true >>>>> >>>>> enabled: true >>>>> >>>>> ipv6: >>>>> >>>>> address: >>>>> >>>>> - ip: fd00::101 >>>>> >>>>> prefix-length: 128 >>>>> >>>>> - ip: fe80::2d55:7c37:8090:7261 >>>>> >>>>> prefix-length: 64 >>>>> >>>>> auto-dns: true >>>>> >>>>> auto-gateway: true >>>>> >>>>> auto-route-table-id: 0 >>>>> >>>>> auto-routes: true >>>>> >>>>> autoconf: true >>>>> >>>>> dhcp: true >>>>> >>>>> enabled: true >>>>> >>>>> lldp: >>>>> >>>>> enabled: false >>>>> >>>>> mac-address: 52:55:00:D1:55:01 >>>>> >>>>> mtu: 1500 >>>>> >>>>> name: eth0 >>>>> >>>>> state: up >>>>> >>>>> type: ethernet >>>>> >>>>> default-gw: >>>>> >>>>> metaInfo: >>>>> >>>>> time: "2021-12-15T13:45:40Z" >>>>> >>>>> state: >>>>> >>>>> routes: >>>>> >>>>> running: >>>>> >>>>> - destination: 0.0.0.0/0 >>>>> >>>>> metric: 100 >>>>> >>>>> next-hop-address: 192.168.66.2 >>>>> >>>>> next-hop-interface: eth0 >>>>> >>>>> table-id: 254 >>>>> >>>>> >>>>> >>>>> Stdout (the desired state to be applied by nmstatectl) >>>>> >>>>> interfaces: >>>>> >>>>> - bridge: >>>>> >>>>> options: >>>>> >>>>> stp: >>>>> >>>>> enabled: false >>>>> >>>>> port: >>>>> >>>>> - name: eth0 >>>>> >>>>> vlan: >>>>> >>>>> mode: trunk >>>>> >>>>> trunk-tags: >>>>> >>>>> - id-range: >>>>> >>>>> max: 4094 >>>>> >>>>> min: 2 >>>>> >>>>> description: Linux bridge with base interface as a port >>>>> >>>>> ipv4: >>>>> >>>>> dhcp: true >>>>> >>>>> enabled: true >>>>> >>>>> mac-address: 52:55:00:D1:55:01 >>>>> >>>>> name: br1 >>>>> >>>>> state: up >>>>> >>>>> type: linux-bridge >>>>> >>>>> >>>>> >>>>> For more details, please review the design doc and share your feedback about it. >>>>> >>>>> Thanks, >>>>> Alona. >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google Groups "kubevirt-dev" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev+unsubscribe(a)googlegroups.com. >>>>> To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/CAJOMPacxNke2pJnjJ1PKxz0f%... . >>>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups "kubevirt-dev" group. >>>> To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev+unsubscribe(a)googlegroups.com. >>>> To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/CAHOEP56e4zhHPtP8rZp%3D%3D... . >>> >>> >>> >>> -- >>> Quique Llorente >>> >>> CNV networking Senior Software Engineer >>> >>> Red Hat EMEA >>> >>> ellorent(a)redhat.com >>> >>> @RedHat Red Hat Red Hat >>>

On Mon, Jan 3, 2022 at 8:42 AM Felix Enrique Llorente Pastora <ellorent(a)redhat.com&gt; wrote: > > > > On Tue, Dec 28, 2021 at 9:40 AM Dan Kenigsberg <danken(a)redhat.com&gt; wrote: >> >> On Sun, Dec 19, 2021 at 11:20 AM Alona Paz <alkaplan(a)redhat.com&gt; wrote: >> > >> > >> > >> > On Sun, Dec 19, 2021 at 10:42 AM Dan Kenigsberg <danken(a)redhat.com&gt; wrote: >> >> >> >> >> >> >> >> On Fri, Dec 17, 2021 at 5:25 PM Felix Enrique Llorente Pastora < ellorent(a)redhat.com&gt; wrote: >> >>> >> >>> >> >>> >> >>> On Fri, Dec 17, 2021 at 3:58 PM Dan Kenigsberg <danken(a)redhat.com&gt; wrote: >> >>>> >> >>>> Thanks for sharing your plan. >> >>>> >> >>>> I would like to create a Linux bond of all the NICs of my machine, create a Linux bridge on top of it, and copy my original mac/IP/routing to that bridge. Can you share a policy that would let me express this desired state? >> >>> >> >>> >> >>> You need to know the number of NICs, for example for three of them this would be it. >> >> >> >> >> >> Thanks. Having a Python-like `0:-1` slice index to select a sublist could be even cooler. >> >> >> >> More importantly, I would like to be able to select all interfaces which have vlan 204 enabled on their switch port, as reported by lldp. What would the capture look like in this case? >> > >> > >> > capture: >> > vlan-204: interfaces.lldp.enabled=="true" | interfaces.lldp.neighbors.ieee-802-1-vlans.vid==204 >> > >> > But I suspect our current implementation will fail resolving this capture since not all the neighbor entries have the "ieee-802-1-vlans" key. >> > We can change the implementation not to fail if at least one of the entries in a list has the required path. >> > >> > Then, even the following capture should work- >> > >> > capture: >> > vlan-204: interfaces.lldp.neighbors.ieee-802-1-vlans.vid==204 >> >> Thanks. I find the capture language very powerful, I think I would >> benefit from a tool that helps me build them, based on the current >> state. >> E.g a command line such as >> nmstatectl show | nmpolicy testcapture >> 'interfaces.lldp.enabled=="true" | >> interfaces.lldp.neighbors.ieee-802-1-vlans.vid==204' >> that would emit the value of yaml-encoded captures. >> >> Would something like this be possible? > > > You can inspect that information at ~/.cache/nmpolicy/cache.yaml, it will dump there the captured states. I see a few drawbacks to relying on this: - caches are typically not visible to end users, their content and location may change - I need to write a full policy; I prefer to focus on a single capture string - When I run the nmpolicyctl I create a side effect (changing the cache). I don't want to do so when I test captures. I'm looking for a pure test-only usage.

P.s. because we keep the cache for a long while, we need to place a version number in it, in case we ever need to make a breaking change in its format. Regards, Dan.