VMs as real hosts on the same network

[mabi]

Hello,

I am trying out VMM on an OpenBSD 6.4 server which has the following network interfaces defined:

[bnx0]+[bnx1]-->[trunk0]-->[vlan2]
[bnx0]+[bnx1]-->[trunk0]-->[vlan6]-->[bridge6]

The vlan2 is for the internal (management) network and vlan6 for the public (internet) network. I manage my server from vlan2 and would like to have my virtual machines on vlan6 which uses public IP addresses. For that purpose I have setup my /etc/hostname.* files as such:

hostname.bnx0 + hostname.bnx1:
up

hostname.trunk0:
trunkproto failover trunkport bnx0 trunkport bnx1 up

hostname.vlan2:
inet 192.168.1.5 255.255.255.0 192.168.1.255 vnetid 2 parent trunk0 description "private"

hostname.vlan6:
vnetid 6 parent trunk0 description "public" up

hostname.bridge6:
add vlan6

I am actually using Option 4 from the Networking chapter in the virtualization FAQ (https://www.openbsd.org/faq/faq16.html) just that my setup has a redundant link (trunk0) and a VLAN (vlan6). So in theory that should work but unfortunately when I start a VM to install OpenBSD 6.4 from the bsd.rd boot file I do not have any network connectivity. I tried with DHCP first and in that case on the DHCP server I see the DHCPDISCOVER and DHCPOFFER requests/answer but there is never a DHCPACK. Then I tried assigning a static IP directly but still no network connectivity. I can't ping the default gateway of that public network. Checking with tcpdump on the firewall I can see the ARP who-has request and the ARP reply back the the VM but again it seems like the VM does not get it.

Here is my vm.conf conf file:

switch "uplink_vlan6" {
interface bridge6
}

vm "example" {
disable
memory 2G
boot "/home/admin/bsd.rd"
disk "/var/vmm/example.qcow2"

interface {
switch "uplink_vlan6"
lladdr fe:e1:bb:01:01:01
}
}

I have also totally disabled pf on that OpenBSD VMM server but that did not change anything (I am using the default pf.conf from the installation)

Any ideas what I might be doing wrong or forgetting?

Regards,
Mabi

[Mischa]

Post by mabi
Hello,
[bnx0]+[bnx1]-->[trunk0]-->[vlan2]
[bnx0]+[bnx1]-->[trunk0]-->[vlan6]-->[bridge6]
up
trunkproto failover trunkport bnx0 trunkport bnx1 up
inet 192.168.1.5 255.255.255.0 192.168.1.255 vnetid 2 parent trunk0 description "private"
vnetid 6 parent trunk0 description "public" up
add vlan6

It might be as easy as adding: up

# cat /etc/hostname.bridge6
add vlan6
up

By default the bridge interface is not brought up.
You can also run: ifconfig bridge6 up

This will most likely be the "problem".

Mischa

Post by mabi
I am actually using Option 4 from the Networking chapter in the virtualization FAQ (https://www.openbsd.org/faq/faq16.html) just that my setup has a redundant link (trunk0) and a VLAN (vlan6). So in theory that should work but unfortunately when I start a VM to install OpenBSD 6.4 from the bsd.rd boot file I do not have any network connectivity. I tried with DHCP first and in that case on the DHCP server I see the DHCPDISCOVER and DHCPOFFER requests/answer but there is never a DHCPACK. Then I tried assigning a static IP directly but still no network connectivity. I can't ping the default gateway of that public network. Checking with tcpdump on the firewall I can see the ARP who-has request and the ARP reply back the the VM but again it seems like the VM does not get it.
switch "uplink_vlan6" {
interface bridge6
}
vm "example" {
disable
memory 2G
boot "/home/admin/bsd.rd"
disk "/var/vmm/example.qcow2"
interface {
switch "uplink_vlan6"
lladdr fe:e1:bb:01:01:01
}
}
I have also totally disabled pf on that OpenBSD VMM server but that did not change anything (I am using the default pf.conf from the installation)
Any ideas what I might be doing wrong or forgetting?
Regards,
Mabi

[mabi]

This post might be inappropriate. Click to display it.

[Mischa]

Post by mabi
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by Mischa
It might be as easy as adding: up
cat /etc/hostname.bridge6
==========================
add vlan6
up
By default the bridge interface is not brought up.
You can also run: ifconfig bridge6 up

Good idea and I added "up" to my hostname.bridge6 file but it looks like it was already up (at least by doing an ifconfig bridge6 shows the "UP" flag). Neverthless to be on the safe side I rebooted the server but still not connectivity on the vlan6/bridge6 network for the VMs.
11:59:35.672258 0.0.0.0.68 > 255.255.255.255.67: xid:0xbafb375b [|bootp] [tos 0x10]
Dec 7 12:00:27 dhcpsrv dhcpd[18917]: DHCPDISCOVER from fe:e1:bb:01:01:01 via XXX.XXX.XXX.1
Dec 7 12:00:27 dhcpsrv dhcpd[18917]: DHCPOFFER on XXX.XXX.XXX.101 to fe:e1:bb:01:01:01 via XXX.XXX.XXX.1
The IP address ending with .1 is the gateway on my public network and the one ending with .101 is the IP which should be assigned to my OpenBSD VM.
It seems like the traffic is not flowing back to the VM itself.
I just found a very interesting behaviour by running tcpdump on pretty much all interfaces of my server to analyze the traffic at different levels and BINGO: as soon as I run tcpdump on my trunk0 interface the DHCP request goes through and my VM has network connectivity! But as soon as I stop tcpdump on the trunk interface: no more network connectivity...
Now as far as I know running tcpdump enables promiscous mode (PROMISC flag on the interface) and this should the reason why it works.
But now what does it mean for my setup, do I need to enable promiscuous mode on my trunk interface manually? and if yes how can I do that?

The VLAN does require an IP address as far as I am aware.

Mischa

[Martin Sukany]

could you post here your /etc/pf.conf rules?

Post by Mischa

Post by mabi
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by Mischa
It might be as easy as adding: up
cat /etc/hostname.bridge6
==========================
add vlan6
up
By default the bridge interface is not brought up.
You can also run: ifconfig bridge6 up

Good idea and I added "up" to my hostname.bridge6 file but it looks like it was already up (at least by doing an ifconfig bridge6 shows the "UP" flag). Neverthless to be on the safe side I rebooted the server but still not connectivity on the vlan6/bridge6 network for the VMs.
11:59:35.672258 0.0.0.0.68 > 255.255.255.255.67: xid:0xbafb375b [|bootp] [tos 0x10]
Dec 7 12:00:27 dhcpsrv dhcpd[18917]: DHCPDISCOVER from fe:e1:bb:01:01:01 via XXX.XXX.XXX.1
Dec 7 12:00:27 dhcpsrv dhcpd[18917]: DHCPOFFER on XXX.XXX.XXX.101 to fe:e1:bb:01:01:01 via XXX.XXX.XXX.1
The IP address ending with .1 is the gateway on my public network and the one ending with .101 is the IP which should be assigned to my OpenBSD VM.
It seems like the traffic is not flowing back to the VM itself.
I just found a very interesting behaviour by running tcpdump on pretty much all interfaces of my server to analyze the traffic at different levels and BINGO: as soon as I run tcpdump on my trunk0 interface the DHCP request goes through and my VM has network connectivity! But as soon as I stop tcpdump on the trunk interface: no more network connectivity...
Now as far as I know running tcpdump enables promiscous mode (PROMISC flag on the interface) and this should the reason why it works.
But now what does it mean for my setup, do I need to enable promiscuous mode on my trunk interface manually? and if yes how can I do that?

The VLAN does require an IP address as far as I am aware.
Mischa

[mabi]

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by Martin Sukany
could you post here your /etc/pf.conf rules?

Sure, it's actually the default OpenBSD 6.4 one as you can see below:

# $OpenBSD: pf.conf,v 1.55 2017/12/03 20:40:04 sthen Exp $
#
# See pf.conf(5) and /etc/examples/pf.conf

set skip on lo

block return log # block stateless traffic
pass # establish keep-state

# By default, do not permit remote connections to X11
block return in on ! lo0 proto tcp to port 6000:6010

# Port build user does not need network
block return out log proto {tcp udp} user _pbuild


See my previous mail answering Mischa, his solution of adding an IP to the VLAN interface solves my issue...

[mabi]

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by Mischa
The VLAN does require an IP address as far as I am aware.

Thanks that worked. I now have network connectivity on my public VM VLAN. I saw that adding an IP to my VLAN interface automatically set the trunk interface to PROMISC.

I was trying to avoid "wasting" an IP address as there is no real need for an IP on that VLAN interface on the server itself. But if that's the only way I am fine with that :)

[Stuart Henderson]

Post by mabi
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by Mischa
The VLAN does require an IP address as far as I am aware.

Thanks that worked. I now have network connectivity on my public VM VLAN. I saw that adding an IP to my VLAN interface automatically set the trunk interface to PROMISC.
I was trying to avoid "wasting" an IP address as there is no real need for an IP on that VLAN interface on the server itself. But if that's the only way I am fine with that :)

That sounds like a bug...