vlan + bridge + xen

Hi

I was helping a friend of mine debug weird issue with
Xen networking the other day.  The setup involves a
dom0 with a single NIC (fxp(4)), vlan(4) interfaces
attached to the fxp in the dom0, and bridged to additional
xennet(4) interfaces in the domU.  Because the xvif/xennet
pair
seems to have a hard 1500 byte limit, the tagged packets
can't
bridged without a drop in MTU, which is undesirable.

Anyway, while from the dom0 the vlan(4) interfaces work
as expected, the connection of the domU's xennet to the
tagged frames on the copper is acting extremely weird.

The domU can ping6 ff02::1%xennetX both itself and the xvif
in the dom0. The MAC table in the dom0's bridge for this
interface
shows the addresses I expect it should, sometimes entries
even
appear in the NDP/ARP tables in remote machines.  But not
so
much as a icmp ping response seems to get received by the
domU.

So, outgoing from the domU seems to be working, but
incoming
seems not to.

All interfaces are marked up, and whatnot.

I tested this on two 4.99.4 Xen3 dom0s,
and a 3.x Xen2 dom0 as well.  All the same.

Any ideas of where to look for clues as to what's causing
this?

	Jonathan Kollasch

On Fri, Mar 30, 2007 at 10:30:12AM -0500, Jonathan A.
Kollasch wrote:
> Hi
> 
> I was helping a friend of mine debug weird issue with
> Xen networking the other day.  The setup involves a
> dom0 with a single NIC (fxp(4)), vlan(4) interfaces
> attached to the fxp in the dom0, and bridged to
additional
> xennet(4) interfaces in the domU.  Because the
xvif/xennet pair
> seems to have a hard 1500 byte limit, the tagged
packets can't
> bridged without a drop in MTU, which is undesirable.
> 
> Anyway, while from the dom0 the vlan(4) interfaces
work
> as expected, the connection of the domU's xennet to
the
> tagged frames on the copper is acting extremely weird.
> 
> The domU can ping6 ff02::1%xennetX both itself and the
xvif
> in the dom0. The MAC table in the dom0's bridge for
this interface
> shows the addresses I expect it should, sometimes
entries even
> appear in the NDP/ARP tables in remote machines.  But
not so
> much as a icmp ping response seems to get received by
the domU.
> 
> So, outgoing from the domU seems to be working, but
incoming
> seems not to.
> 
> All interfaces are marked up, and whatnot.
> 
> I tested this on two 4.99.4 Xen3 dom0s,
> and a 3.x Xen2 dom0 as well.  All the same.
> 
> Any ideas of where to look for clues as to what's
causing this?

Yes: 802.1Q packets are forwarded to the vlan(4) layer and
so the bridge
never see them. This just won't work, the solution is to use
one
vif per vlan.

-- 
Manuel Bouyer <bouyerantioche.eu.org>
     NetBSD: 26 ans d'experience feront toujours la
difference
--

On Fri, Mar 30, 2007 at 10:30:12AM -0500, Jonathan A.
Kollasch wrote:
> Hi
> 
> I was helping a friend of mine debug weird issue with
> Xen networking the other day.  The setup involves a
> dom0 with a single NIC (fxp(4)), vlan(4) interfaces
> attached to the fxp in the dom0, and bridged to
additional
> xennet(4) interfaces in the domU.  Because the
xvif/xennet pair
> seems to have a hard 1500 byte limit, the tagged
packets can't
> bridged without a drop in MTU, which is undesirable.
> 
> Anyway, while from the dom0 the vlan(4) interfaces
work
> as expected, the connection of the domU's xennet to
the
> tagged frames on the copper is acting extremely weird.
> 
> The domU can ping6 ff02::1%xennetX both itself and the
xvif
> in the dom0. The MAC table in the dom0's bridge for
this interface
> shows the addresses I expect it should, sometimes
entries even
> appear in the NDP/ARP tables in remote machines.  But
not so
> much as a icmp ping response seems to get received by
the domU.
> 
> So, outgoing from the domU seems to be working, but
incoming
> seems not to.
> 
> All interfaces are marked up, and whatnot.
> 
> I tested this on two 4.99.4 Xen3 dom0s,
> and a 3.x Xen2 dom0 as well.  All the same.
> 
> Any ideas of where to look for clues as to what's
causing this?

Yes: 802.1Q packets are forwarded to the vlan(4) layer and
so the bridge
never see them. This just won't work, the solution is to use
one
vif per vlan.

-- 
Manuel Bouyer <bouyerantioche.eu.org>
     NetBSD: 26 ans d'experience feront toujours la
difference
--

On Fri, Mar 30, 2007 at 08:27:50PM +0200, Manuel Bouyer
wrote:
> Yes: 802.1Q packets are forwarded to the vlan(4) layer
and so the bridge

Ok, makes sense. Somehow I have this vague recollection of
that working,
(or maybe i just tried it and it didn't work) but that's
besides the point.

> never see them. This just won't work, the solution is
to use one
> vif per vlan.

If only that actually worked.  :l

	Jonathan Kollasch

On Fri, Mar 30, 2007 at 08:27:50PM +0200, Manuel Bouyer
wrote:
> Yes: 802.1Q packets are forwarded to the vlan(4) layer
and so the bridge

Ok, makes sense. Somehow I have this vague recollection of
that working,
(or maybe i just tried it and it didn't work) but that's
besides the point.

> never see them. This just won't work, the solution is
to use one
> vif per vlan.

If only that actually worked.  :l

	Jonathan Kollasch

On Fri, Mar 30, 2007 at 08:27:50PM +0200, Manuel Bouyer
wrote:
> On Fri, Mar 30, 2007 at 10:30:12AM -0500, Jonathan A.
Kollasch wrote:
> > Hi
> > 
> > I was helping a friend of mine debug weird issue
with
> > Xen networking the other day.  The setup involves
a
> > dom0 with a single NIC (fxp(4)), vlan(4)
interfaces
> > attached to the fxp in the dom0, and bridged to
additional
> > xennet(4) interfaces in the domU.  Because the
xvif/xennet pair
> > seems to have a hard 1500 byte limit, the tagged
packets can't
> > bridged without a drop in MTU, which is
undesirable.
> > 
> > Anyway, while from the dom0 the vlan(4) interfaces
work
> > as expected, the connection of the domU's xennet
to the
> > tagged frames on the copper is acting extremely
weird.
> > 
> > The domU can ping6 ff02::1%xennetX both itself and
the xvif
> > in the dom0. The MAC table in the dom0's bridge
for this interface
> > shows the addresses I expect it should, sometimes
entries even
> > appear in the NDP/ARP tables in remote machines. 
But not so
> > much as a icmp ping response seems to get received
by the domU.
> > 
> > So, outgoing from the domU seems to be working,
but incoming
> > seems not to.
> > 
> > All interfaces are marked up, and whatnot.
> > 
> > I tested this on two 4.99.4 Xen3 dom0s,
> > and a 3.x Xen2 dom0 as well.  All the same.
> > 
> > Any ideas of where to look for clues as to what's
causing this?
> 
> Yes: 802.1Q packets are forwarded to the vlan(4) layer
and so the bridge
> never see them. This just won't work, the solution is
to use one
> vif per vlan.

Hi, friend of jakllsch here.  My weird networking issue.

I currently have one vif per vlan.  The issue i was having
is
slightly different than jakllsch's, in that I was trying to
communicate using domUs on two different dom0s.  on both
the
dom0s, i have an fxp card, with a vlan interface - vlan216. 
on
each dom0, i add vlan216 to bridge0, then associate the
domU's
xennet device with bridge0.  And now, a poor attempt at
ascii
art:

                         +-------+
                         | dom0a |
                         +-------+
                            fxp0
+-------+                     |
| domUa |xennet0--bridge0--vlan216-----
+-------+                            +----------+
                                     |  switch  |
+-------+                            +----------+
| domUb |xennet0--bridge0--vlan216------/
+-------+                    |
                           fxp0
                         +-------+
                         | dom0b |
                         +-------+


i don't know, does this make sense? have i just embarassed
myself
by showing the universe that i am not only no artist, but i
don't
know how bridges and vlans work?  alas.  this is why i
asked
jonathan to send the initial mail, i'll just make a hash of
it.

if i put an IP on dom0a's vlan216 interface, it can ping
domUb
without trouble.  when trying to ping domUb from domUa, the
ping
is observed at domUa's xennet0 device, and the reply is
observed
as far up the chain as the vlan216 interface of dom0a, but
is not
seen anywhere on domUb or dom0b.
  --david

> -- 
> Manuel Bouyer <bouyerantioche.eu.org>
>      NetBSD: 26 ans d'experience feront toujours la
difference
> --

On Fri, Mar 30, 2007 at 08:27:50PM +0200, Manuel Bouyer
wrote:
> On Fri, Mar 30, 2007 at 10:30:12AM -0500, Jonathan A.
Kollasch wrote:
> > Hi
> > 
> > I was helping a friend of mine debug weird issue
with
> > Xen networking the other day.  The setup involves
a
> > dom0 with a single NIC (fxp(4)), vlan(4)
interfaces
> > attached to the fxp in the dom0, and bridged to
additional
> > xennet(4) interfaces in the domU.  Because the
xvif/xennet pair
> > seems to have a hard 1500 byte limit, the tagged
packets can't
> > bridged without a drop in MTU, which is
undesirable.
> > 
> > Anyway, while from the dom0 the vlan(4) interfaces
work
> > as expected, the connection of the domU's xennet
to the
> > tagged frames on the copper is acting extremely
weird.
> > 
> > The domU can ping6 ff02::1%xennetX both itself and
the xvif
> > in the dom0. The MAC table in the dom0's bridge
for this interface
> > shows the addresses I expect it should, sometimes
entries even
> > appear in the NDP/ARP tables in remote machines. 
But not so
> > much as a icmp ping response seems to get received
by the domU.
> > 
> > So, outgoing from the domU seems to be working,
but incoming
> > seems not to.
> > 
> > All interfaces are marked up, and whatnot.
> > 
> > I tested this on two 4.99.4 Xen3 dom0s,
> > and a 3.x Xen2 dom0 as well.  All the same.
> > 
> > Any ideas of where to look for clues as to what's
causing this?
> 
> Yes: 802.1Q packets are forwarded to the vlan(4) layer
and so the bridge
> never see them. This just won't work, the solution is
to use one
> vif per vlan.

Hi, friend of jakllsch here.  My weird networking issue.

I currently have one vif per vlan.  The issue i was having
is
slightly different than jakllsch's, in that I was trying to
communicate using domUs on two different dom0s.  on both
the
dom0s, i have an fxp card, with a vlan interface - vlan216. 
on
each dom0, i add vlan216 to bridge0, then associate the
domU's
xennet device with bridge0.  And now, a poor attempt at
ascii
art:

                         +-------+
                         | dom0a |
                         +-------+
                            fxp0
+-------+                     |
| domUa |xennet0--bridge0--vlan216-----
+-------+                            +----------+
                                     |  switch  |
+-------+                            +----------+
| domUb |xennet0--bridge0--vlan216------/
+-------+                    |
                           fxp0
                         +-------+
                         | dom0b |
                         +-------+


i don't know, does this make sense? have i just embarassed
myself
by showing the universe that i am not only no artist, but i
don't
know how bridges and vlans work?  alas.  this is why i
asked
jonathan to send the initial mail, i'll just make a hash of
it.

if i put an IP on dom0a's vlan216 interface, it can ping
domUb
without trouble.  when trying to ping domUb from domUa, the
ping
is observed at domUa's xennet0 device, and the reply is
observed
as far up the chain as the vlan216 interface of dom0a, but
is not
seen anywhere on domUb or dom0b.
  --david

> -- 
> Manuel Bouyer <bouyerantioche.eu.org>
>      NetBSD: 26 ans d'experience feront toujours la
difference
> --

On Sun, Apr 01, 2007 at 12:07:28PM -0400, Thor Lancelot
Simon wrote:
> On Fri, Mar 30, 2007 at 03:07:35PM -0500, Jonathan A.
Kollasch wrote:
> > On Fri, Mar 30, 2007 at 08:27:50PM +0200, Manuel
Bouyer wrote:
> > > Yes: 802.1Q packets are forwarded to the
vlan(4) layer and so the bridge
> > 
> > Ok, makes sense. Somehow I have this vague
recollection of that working,
> > (or maybe i just tried it and it didn't work) but
that's besides the point.
> > 
> > > never see them. This just won't work, the
solution is to use one
> > > vif per vlan.
> > 
> > If only that actually worked.  :l
> 
> It seems to work fine for me.

It does for me as well. In fact most of my Xen systems use
802.1Q connectivity.

-- 
Manuel Bouyer <bouyerantioche.eu.org>
     NetBSD: 26 ans d'experience feront toujours la
difference
--

On Sun, Apr 01, 2007 at 12:07:28PM -0400, Thor Lancelot
Simon wrote:
> On Fri, Mar 30, 2007 at 03:07:35PM -0500, Jonathan A.
Kollasch wrote:
> > On Fri, Mar 30, 2007 at 08:27:50PM +0200, Manuel
Bouyer wrote:
> > > Yes: 802.1Q packets are forwarded to the
vlan(4) layer and so the bridge
> > 
> > Ok, makes sense. Somehow I have this vague
recollection of that working,
> > (or maybe i just tried it and it didn't work) but
that's besides the point.
> > 
> > > never see them. This just won't work, the
solution is to use one
> > > vif per vlan.
> > 
> > If only that actually worked.  :l
> 
> It seems to work fine for me.

It does for me as well. In fact most of my Xen systems use
802.1Q connectivity.

-- 
Manuel Bouyer <bouyerantioche.eu.org>
     NetBSD: 26 ans d'experience feront toujours la
difference
--

Manuel Bouyer wrote:
> On Sun, Apr 01, 2007 at 12:07:28PM -0400, Thor Lancelot
Simon wrote:
>> On Fri, Mar 30, 2007 at 03:07:35PM -0500, Jonathan
A. Kollasch wrote:
>>> On Fri, Mar 30, 2007 at 08:27:50PM +0200,
Manuel Bouyer wrote:
>>>> Yes: 802.1Q packets are forwarded to the
vlan(4) layer and so the bridge
>>> Ok, makes sense. Somehow I have this vague
recollection of that working,
>>> (or maybe i just tried it and it didn't work)
but that's besides the point.
>>>
>>>> never see them. This just won't work, the
solution is to use one
>>>> vif per vlan.
>>> If only that actually worked.  :l
>> It seems to work fine for me.
> 
> It does for me as well. In fact most of my Xen systems
use 802.1Q connectivity.
> 


As you've probably seen in the thread "boot hangs at
uhci1," moving this 
computer has caused everything to break.

Except this!  My VLANs seem to be happy, and I can
communicate between 
domUs on different dom0s on VLANs without any trouble.
   --david