Network trouble with bridges

Hello together,

I'm running NetBSD 3.0 along with xen2, and today I ran
into trouble  
with my setup.

Up to now, I had dom0 export a disk via nfs to the domUs. I
now  
charged one of the domU's with being the nfs-server.
I have two bridges configured, bridge0 shares an external
wm0 with  
all domUs and bridge1 creates a private lan with dom0's
tap0.

Traffic passing from other physical machines passes just
fine to the  
domU's, but anything between 2 domUs has a delay of about
20 seconds.  
ICMP works, but does not even show the latency, instead
reportint an  
RTT of 0,4ms.
The delay even happens when pinging the domU by itself

Sadly, nfs and everything else now times out with

mount_nfs: bad MNT RPC: RPC: Timed out

This even happens with the GENERIC XEN0 Kernel.
I have ruled out the MACs being the troublemakers, all begin
with aa.

Any clues?

Thanks in advance,
Andreas

On Sun, May 14, 2006 at 09:03:42PM +0200, Andreas Neth
wrote:
> Hello together,
> 
> I'm running NetBSD 3.0 along with xen2, and today I
ran into trouble  
> with my setup.
> 
> Up to now, I had dom0 export a disk via nfs to the
domUs. I now  
> charged one of the domU's with being the nfs-server.
> I have two bridges configured, bridge0 shares an
external wm0 with  
> all domUs and bridge1 creates a private lan with
dom0's tap0.

Is dom0's tap0 used for something ? You don't need to have
an interface
belonging to dom0 for bridging domUs together.

> 
> Traffic passing from other physical machines passes
just fine to the  
> domU's, but anything between 2 domUs has a delay of
about 20 seconds.  
> ICMP works, but does not even show the latency, instead
reportint an  
> RTT of 0,4ms.
> The delay even happens when pinging the domU by itself

You could run tcdump on dom0, to get more details on what's
going on

-- 
Manuel Bouyer <bouyerantioche.eu.org>
     NetBSD: 26 ans d'experience feront toujours la
difference
--

On Sun, May 14, 2006 at 09:03:42PM +0200, Andreas Neth
wrote:
> Hello together,
> 
> I'm running NetBSD 3.0 along with xen2, and today I
ran into trouble  
> with my setup.
> 
> Up to now, I had dom0 export a disk via nfs to the
domUs. I now  
> charged one of the domU's with being the nfs-server.
> I have two bridges configured, bridge0 shares an
external wm0 with  
> all domUs and bridge1 creates a private lan with
dom0's tap0.

Is dom0's tap0 used for something ? You don't need to have
an interface
belonging to dom0 for bridging domUs together.

> 
> Traffic passing from other physical machines passes
just fine to the  
> domU's, but anything between 2 domUs has a delay of
about 20 seconds.  
> ICMP works, but does not even show the latency, instead
reportint an  
> RTT of 0,4ms.
> The delay even happens when pinging the domU by itself

You could run tcdump on dom0, to get more details on what's
going on

-- 
Manuel Bouyer <bouyerantioche.eu.org>
     NetBSD: 26 ans d'experience feront toujours la
difference
--

On 14.05.2006, at 21:35, Manuel Bouyer wrote:

> On Sun, May 14, 2006 at 09:03:42PM +0200, Andreas Neth
wrote:
>>
>> I'm running NetBSD 3.0 along with xen2, and today
I ran into trouble
>> with my setup.
>>
>> Up to now, I had dom0 export a disk via nfs to the
domUs. I now
>> charged one of the domU's with being the
nfs-server.
>> I have two bridges configured, bridge0 shares an
external wm0 with
>> all domUs and bridge1 creates a private lan with
dom0's tap0.
>
> Is dom0's tap0 used for something ? You don't need to
have an  
> interface
> belonging to dom0 for bridging domUs together.

tap0 is/was used for dhcpd

>> Traffic passing from other physical machines passes
just fine to the
>> domU's, but anything between 2 domUs has a delay
of about 20 seconds.
>> ICMP works, but does not even show the latency,
instead reportint an
>> RTT of 0,4ms.
>> The delay even happens when pinging the domU by
itself
>
> You could run tcdump on dom0, to get more details on
what's going on

I already did that. Apart from the delay (which does not
happen  
always from every machine), I can't see anything unusual.

The packets for the nfs-mount also pass by just fine. (debug
mount on  
the nfs-host also shows a successful mount)

On 14.05.2006, at 21:35, Manuel Bouyer wrote:

> On Sun, May 14, 2006 at 09:03:42PM +0200, Andreas Neth
wrote:
>>
>> I'm running NetBSD 3.0 along with xen2, and today
I ran into trouble
>> with my setup.
>>
>> Up to now, I had dom0 export a disk via nfs to the
domUs. I now
>> charged one of the domU's with being the
nfs-server.
>> I have two bridges configured, bridge0 shares an
external wm0 with
>> all domUs and bridge1 creates a private lan with
dom0's tap0.
>
> Is dom0's tap0 used for something ? You don't need to
have an  
> interface
> belonging to dom0 for bridging domUs together.

tap0 is/was used for dhcpd

>> Traffic passing from other physical machines passes
just fine to the
>> domU's, but anything between 2 domUs has a delay
of about 20 seconds.
>> ICMP works, but does not even show the latency,
instead reportint an
>> RTT of 0,4ms.
>> The delay even happens when pinging the domU by
itself
>
> You could run tcdump on dom0, to get more details on
what's going on

I already did that. Apart from the delay (which does not
happen  
always from every machine), I can't see anything unusual.

The packets for the nfs-mount also pass by just fine. (debug
mount on  
the nfs-host also shows a successful mount)

On Sun, May 14, 2006 at 10:28:16PM +0200, Andreas Neth
wrote:
> 
> >>Traffic passing from other physical machines
passes just fine to the
> >>domU's, but anything between 2 domUs has a
delay of about 20 seconds.
> >>ICMP works, but does not even show the latency,
instead reportint an
> >>RTT of 0,4ms.
> >>The delay even happens when pinging the domU by
itself
> >
> >You could run tcdump on dom0, to get more details
on what's going on
> 
> I already did that. Apart from the delay (which does
not happen  
> always from every machine), I can't see anything
unusual.

What would be interesting to know if what is delayed, and if
it's
delayed the same way on the different xvif interfaces (you
may need to
run 2 tcpdump for that)

-- 
Manuel Bouyer <bouyerantioche.eu.org>
     NetBSD: 26 ans d'experience feront toujours la
difference
--

On Sun, May 14, 2006 at 10:28:16PM +0200, Andreas Neth
wrote:
> 
> >>Traffic passing from other physical machines
passes just fine to the
> >>domU's, but anything between 2 domUs has a
delay of about 20 seconds.
> >>ICMP works, but does not even show the latency,
instead reportint an
> >>RTT of 0,4ms.
> >>The delay even happens when pinging the domU by
itself
> >
> >You could run tcdump on dom0, to get more details
on what's going on
> 
> I already did that. Apart from the delay (which does
not happen  
> always from every machine), I can't see anything
unusual.

What would be interesting to know if what is delayed, and if
it's
delayed the same way on the different xvif interfaces (you
may need to
run 2 tcpdump for that)

-- 
Manuel Bouyer <bouyerantioche.eu.org>
     NetBSD: 26 ans d'experience feront toujours la
difference
--

On 14.05.2006, at 22:33, Manuel Bouyer wrote:

> On Sun, May 14, 2006 at 10:28:16PM +0200, Andreas Neth
wrote:
>>
>>>> Traffic passing from other physical
machines passes just fine to  
>>>> the
>>>> domU's, but anything between 2 domUs has a
delay of about 20  
>>>> seconds.
>>>> ICMP works, but does not even show the
latency, instead  
>>>> reportint an
>>>> RTT of 0,4ms.
>>>> The delay even happens when pinging the
domU by itself
>>>
>>> You could run tcdump on dom0, to get more
details on what's going on
>>
>> I already did that. Apart from the delay (which
does not happen
>> always from every machine), I can't see anything
unusual.
>
> What would be interesting to know if what is delayed,
and if it's
> delayed the same way on the different xvif interfaces
(you may need to
> run 2 tcpdump for that)

hope the following dumps illustrate it better.

ok, isis ist the nfsd-domain, and seshat is the nfs-client:

23:32:53.590403 IP isis.1020 > seshat868: UDP, length: 68
23:32:53.590435 IP seshat > isis: icmp 36: seshat udp
port 868  
unreachable

...is something new I logged with tcpdump from seshat on
xennet0

23:33:15.630261 IP isis.1020 > seshat.865: UDP, length:
68
23:33:51.660212 IP seshat.864 > isis.sunrpc: UDP, length:
56
23:33:51.662714 IP isis.sunrpc > seshat.864: UDP, length:
28
23:33:51.662928 IP seshat.863 > isis.sunrpc: UDP, length:
56
23:33:51.663464 IP isis.sunrpc > seshat.863: UDP, length:
28
23:33:51.663706 IP seshat.862 > isis.1020: UDP, length:
116


dom0 logs on seshats' xvif7.0: (ethereal exported)

No.     Time        Source                Destination       
    
Protocol Info
      15 16.572887   isis               seshat            
RPC       
Continuation
      29 28.369415   seshat             isis              
Portmap   
V2 GETPORT Call (Reply In 30) NFS(100003) V:3 UDP
      30 28.371812   isis               seshat            
Portmap   
V2 GETPORT Reply (Call In 29) Port:2049
      31 28.372118   seshat             isis              
Portmap   
V2 GETPORT Call (Reply In 32) MOUNT(100005) V:3 UDP
      32 28.372709   isis               seshat            
Portmap   
V2 GETPORT Reply (Call In 31) Port:1020
      33 28.373060   seshat             isis              
MOUNT     
V3 MNT Call (Reply In 62)[Short Frame]
      46 38.465266   seshat             isis              
Portmap   
V2 GETPORT Call (Reply In 47) NFS(100003) V:3 UDP
      47 38.467642   isis               seshat            
Portmap   
V2 GETPORT Reply (Call In 46) Port:2049
      48 38.467948   seshat             isis              
Portmap   
V2 GETPORT Call (Reply In 49) MOUNT(100005) V:3 UDP
      49 38.470260   isis               seshat            
Portmap   
V2 GETPORT Reply (Call In 48) Port:1020
      50 38.470609   seshat             isis              
MOUNT     
V3 MNT Call (Reply In 104)[Short Frame]
      62 50.413954   isis               seshat            
MOUNT     
V3 MNT Reply (Call In 33)[Short Frame]
     104 72.452883   isis               seshat            
MOUNT     
V3 MNT Reply (Call In 50)[Short Frame]
     148 108.482685  seshat             isis              
Portmap   
V2 GETPORT Call (Reply In 149) NFS(100003) V:3 UDP
     149 108.485102  isis               seshat            
Portmap   
V2 GETPORT Reply (Call In 148) Port:2049
     150 108.485385  seshat             isis              
Portmap   
V2 GETPORT Call (Reply In 151) MOUNT(100005) V:3 UDP
     151 108.485846  isis               seshat            
Portmap   
V2 GETPORT Reply (Call In 150) Port:1020
     152 108.486157  seshat             isis              
MOUNT     
V3 MNT Call (Reply In 183)[Short Frame]


dom0 logs on isis' xvif5.0: (ethereal exported)

note the delayed mount calls (19, 43

No.     Time        Source                Destination       
    
Protocol Info
       1 0.000000    isis              seshat            
RPC       
Continuation
      15 11.796553   seshat            isis              
Portmap  V2  
GETPORT Call (Reply In 16) NFS(100003) V:3 UDP
      16 11.798925   isis              seshat            
Portmap  V2  
GETPORT Reply (Call In 15) Port:2049
      17 11.799253   seshat            isis              
Portmap  V2  
GETPORT Call (Reply In 18) MOUNT(100005) V:3 UDP
      18 11.799825   isis              seshat            
Portmap  V2  
GETPORT Reply (Call In 17) Port:1020
      19 11.800195   seshat            isis              
MOUNT    V3  
MNT Call (Reply In 62)[Short Frame]
      39 21.892404   seshat            isis              
Portmap  V2  
GETPORT Call (Reply In 40) NFS(100003) V:3 UDP
      40 21.894755   isis              seshat            
Portmap  V2  
GETPORT Reply (Call In 39) Port:2049
      41 21.895083   seshat            isis              
Portmap  V2  
GETPORT Call (Reply In 42) MOUNT(100005) V:3 UDP
      42 21.897373   isis              seshat            
Portmap  V2  
GETPORT Reply (Call In 41) Port:1020
      43 21.897744   seshat            isis              
MOUNT    V3  
MNT Call (Reply In 100)[Short Frame]
      62 33.841068   isis              seshat            
MOUNT    V3  
MNT Reply (Call In 19)[Short Frame]
     100 55.879996   isis              seshat            
MOUNT    V3  
MNT Reply (Call In 43)[Short Frame]
     144 91.909822   seshat            isis              
Portmap  V2  
GETPORT Call (Reply In 145) NFS(100003) V:3 UDP
     145 91.912216   isis              seshat            
Portmap  V2  
GETPORT Reply (Call In 144) Port:2049
     146 91.912518   seshat            isis              
Portmap  V2  
GETPORT Call (Reply In 147) MOUNT(100005) V:3 UDP
     147 91.912960   isis              seshat            
Portmap  V2  
GETPORT Reply (Call In 146) Port:1020
     148 91.913290   seshat            isis              
MOUNT    V3  
MNT Call (Reply In 183)[Short Frame]
     183 113.949976  isis              seshat            
MOUNT    V3  
MNT Reply (Call In 148)[Short Frame]


ping from seshat to isis is currently without delay,
vice-versa is  
delayed.

seshat# ping isis
PING isis (isis ): 56 data bytes
64 bytes from isis : icmp_seq=0 ttl=255 time=0.280 ms
64 bytes from isis : icmp_seq=1 ttl=255 time=0.250 ms
64 bytes from isis : icmp_seq=2 ttl=255 time=0.265 ms

isis# ping seshat
PING seshat (seshat): 56 data bytes
.....(imagine 20secs delay here)
64 bytes from seshat: icmp_seq=0 ttl=255 time=45.535 ms
64 bytes from seshat: icmp_seq=1 ttl=255 time=0.332 ms
64 bytes from seshat: icmp_seq=2 ttl=255 time=0.323 ms

On 14.05.2006, at 22:33, Manuel Bouyer wrote:

> On Sun, May 14, 2006 at 10:28:16PM +0200, Andreas Neth
wrote:
>>
>>>> Traffic passing from other physical
machines passes just fine to  
>>>> the
>>>> domU's, but anything between 2 domUs has a
delay of about 20  
>>>> seconds.
>>>> ICMP works, but does not even show the
latency, instead  
>>>> reportint an
>>>> RTT of 0,4ms.
>>>> The delay even happens when pinging the
domU by itself
>>>
>>> You could run tcdump on dom0, to get more
details on what's going on
>>
>> I already did that. Apart from the delay (which
does not happen
>> always from every machine), I can't see anything
unusual.
>
> What would be interesting to know if what is delayed,
and if it's
> delayed the same way on the different xvif interfaces
(you may need to
> run 2 tcpdump for that)

hope the following dumps illustrate it better.

ok, isis ist the nfsd-domain, and seshat is the nfs-client:

23:32:53.590403 IP isis.1020 > seshat868: UDP, length: 68
23:32:53.590435 IP seshat > isis: icmp 36: seshat udp
port 868  
unreachable

...is something new I logged with tcpdump from seshat on
xennet0

23:33:15.630261 IP isis.1020 > seshat.865: UDP, length:
68
23:33:51.660212 IP seshat.864 > isis.sunrpc: UDP, length:
56
23:33:51.662714 IP isis.sunrpc > seshat.864: UDP, length:
28
23:33:51.662928 IP seshat.863 > isis.sunrpc: UDP, length:
56
23:33:51.663464 IP isis.sunrpc > seshat.863: UDP, length:
28
23:33:51.663706 IP seshat.862 > isis.1020: UDP, length:
116


dom0 logs on seshats' xvif7.0: (ethereal exported)

No.     Time        Source                Destination       
    
Protocol Info
      15 16.572887   isis               seshat            
RPC       
Continuation
      29 28.369415   seshat             isis              
Portmap   
V2 GETPORT Call (Reply In 30) NFS(100003) V:3 UDP
      30 28.371812   isis               seshat            
Portmap   
V2 GETPORT Reply (Call In 29) Port:2049
      31 28.372118   seshat             isis              
Portmap   
V2 GETPORT Call (Reply In 32) MOUNT(100005) V:3 UDP
      32 28.372709   isis               seshat            
Portmap   
V2 GETPORT Reply (Call In 31) Port:1020
      33 28.373060   seshat             isis              
MOUNT     
V3 MNT Call (Reply In 62)[Short Frame]
      46 38.465266   seshat             isis              
Portmap   
V2 GETPORT Call (Reply In 47) NFS(100003) V:3 UDP
      47 38.467642   isis               seshat            
Portmap   
V2 GETPORT Reply (Call In 46) Port:2049
      48 38.467948   seshat             isis              
Portmap   
V2 GETPORT Call (Reply In 49) MOUNT(100005) V:3 UDP
      49 38.470260   isis               seshat            
Portmap   
V2 GETPORT Reply (Call In 48) Port:1020
      50 38.470609   seshat             isis              
MOUNT     
V3 MNT Call (Reply In 104)[Short Frame]
      62 50.413954   isis               seshat            
MOUNT     
V3 MNT Reply (Call In 33)[Short Frame]
     104 72.452883   isis               seshat            
MOUNT     
V3 MNT Reply (Call In 50)[Short Frame]
     148 108.482685  seshat             isis              
Portmap   
V2 GETPORT Call (Reply In 149) NFS(100003) V:3 UDP
     149 108.485102  isis               seshat            
Portmap   
V2 GETPORT Reply (Call In 148) Port:2049
     150 108.485385  seshat             isis              
Portmap   
V2 GETPORT Call (Reply In 151) MOUNT(100005) V:3 UDP
     151 108.485846  isis               seshat            
Portmap   
V2 GETPORT Reply (Call In 150) Port:1020
     152 108.486157  seshat             isis              
MOUNT     
V3 MNT Call (Reply In 183)[Short Frame]


dom0 logs on isis' xvif5.0: (ethereal exported)

note the delayed mount calls (19, 43

No.     Time        Source                Destination       
    
Protocol Info
       1 0.000000    isis              seshat            
RPC       
Continuation
      15 11.796553   seshat            isis              
Portmap  V2  
GETPORT Call (Reply In 16) NFS(100003) V:3 UDP
      16 11.798925   isis              seshat            
Portmap  V2  
GETPORT Reply (Call In 15) Port:2049
      17 11.799253   seshat            isis              
Portmap  V2  
GETPORT Call (Reply In 18) MOUNT(100005) V:3 UDP
      18 11.799825   isis              seshat            
Portmap  V2  
GETPORT Reply (Call In 17) Port:1020
      19 11.800195   seshat            isis              
MOUNT    V3  
MNT Call (Reply In 62)[Short Frame]
      39 21.892404   seshat            isis              
Portmap  V2  
GETPORT Call (Reply In 40) NFS(100003) V:3 UDP
      40 21.894755   isis              seshat            
Portmap  V2  
GETPORT Reply (Call In 39) Port:2049
      41 21.895083   seshat            isis              
Portmap  V2  
GETPORT Call (Reply In 42) MOUNT(100005) V:3 UDP
      42 21.897373   isis              seshat            
Portmap  V2  
GETPORT Reply (Call In 41) Port:1020
      43 21.897744   seshat            isis              
MOUNT    V3  
MNT Call (Reply In 100)[Short Frame]
      62 33.841068   isis              seshat            
MOUNT    V3  
MNT Reply (Call In 19)[Short Frame]
     100 55.879996   isis              seshat            
MOUNT    V3  
MNT Reply (Call In 43)[Short Frame]
     144 91.909822   seshat            isis              
Portmap  V2  
GETPORT Call (Reply In 145) NFS(100003) V:3 UDP
     145 91.912216   isis              seshat            
Portmap  V2  
GETPORT Reply (Call In 144) Port:2049
     146 91.912518   seshat            isis              
Portmap  V2  
GETPORT Call (Reply In 147) MOUNT(100005) V:3 UDP
     147 91.912960   isis              seshat            
Portmap  V2  
GETPORT Reply (Call In 146) Port:1020
     148 91.913290   seshat            isis              
MOUNT    V3  
MNT Call (Reply In 183)[Short Frame]
     183 113.949976  isis              seshat            
MOUNT    V3  
MNT Reply (Call In 148)[Short Frame]


ping from seshat to isis is currently without delay,
vice-versa is  
delayed.

seshat# ping isis
PING isis (isis ): 56 data bytes
64 bytes from isis : icmp_seq=0 ttl=255 time=0.280 ms
64 bytes from isis : icmp_seq=1 ttl=255 time=0.250 ms
64 bytes from isis : icmp_seq=2 ttl=255 time=0.265 ms

isis# ping seshat
PING seshat (seshat): 56 data bytes
.....(imagine 20secs delay here)
64 bytes from seshat: icmp_seq=0 ttl=255 time=45.535 ms
64 bytes from seshat: icmp_seq=1 ttl=255 time=0.332 ms
64 bytes from seshat: icmp_seq=2 ttl=255 time=0.323 ms

On Sun, May 14, 2006 at 11:08:16PM +0200, Andreas Neth
wrote:
> [...]
> ping from seshat to isis is currently without delay,
vice-versa is  
> delayed.
> 
> seshat# ping isis
> PING isis (isis ): 56 data bytes
> 64 bytes from isis : icmp_seq=0 ttl=255 time=0.280 ms
> 64 bytes from isis : icmp_seq=1 ttl=255 time=0.250 ms
> 64 bytes from isis : icmp_seq=2 ttl=255 time=0.265 ms
> 
> isis# ping seshat
> PING seshat (seshat): 56 data bytes
> .....(imagine 20secs delay here)
> 64 bytes from seshat: icmp_seq=0 ttl=255 time=45.535 ms
> 64 bytes from seshat: icmp_seq=1 ttl=255 time=0.332 ms
> 64 bytes from seshat: icmp_seq=2 ttl=255 time=0.323 ms

There doesn't seem to be packet loss, nor delay once ping
starts
transmitting. Could you try to ktrace ping to see what it's
doing in these
20 seconds ? Could it be a DNS issue ?

-- 
Manuel Bouyer <bouyerantioche.eu.org>
     NetBSD: 26 ans d'experience feront toujours la
difference
--