Critical errors: Connection between MSM-B mother board port X ..

Hello,

I'm having some serious problems with my blackdiamond switch
and I don't
know how to proceed, because I cannot afford any downtime at
the moment.
I can get a spare BD6808 switch and spare MSM slots if
needed.

I'm running

Platform:         BD6808

License:          Full L3
System Mode:      802.1Q EtherType is 8100 (Hex).
Diag Mode:        Fast Post
RED Probability:  0
DLCS:             Disabled
Backplane Ls:     port-based

SysHealth Check:  Enabled.   Alarm Level = Log
Recovery Mode:    None
Transceiver Diag: Enabled.    Failure action:  log only
Fdb-Scan Diag:    Enabled.    Failure action:  log only
MSM Failover:     take-links-down

MSM:              MSM-A        MSM-B        
                  ------------ ------------ 
Module Type:      MSM64i       MSM64i       
Current State:    MASTER       SLAVE        

Default Cfg Mode: Standard     
Image Selected:   Primary      Primary      
Image Booted:     Primary                   
Config Selected:  Primary      Primary      
Config Booted:    Primary                   

EW Versions:      
                  MSM-A        
                  ------------ 
Primary EW Ver:   7.3.3.4 [non-ssh] [wlan] 
Secondary EW Ver: 7.3.3.4 [non-ssh] [wlan] 

                  MSM-B        
                  ------------ 
Primary EW Ver:   7.3.3.4 [non-ssh] [wlan] 
Secondary EW Ver: 7.3.3.4 [non-ssh] [wlan] 

and in syslog I am receiving messages like this>>

....
07/12/2007 15:00:49.23 <Info:SYST> MSM-B is inserted.
07/12/2007 15:00:03.23 <Erro:SYST> MSM-B may have
problem to boot.
Reboot it.
07/12/2007 14:58:23.23 <Erro:SYST> MSM-B may have
problem to boot.
Reboot it.
07/12/2007 14:57:44.19 <Info:SYST> slave MSM
communications pipe is
closed 
07/12/2007 14:57:44.15 <Info:SYST> Removed MSM-B base
module
07/12/2007 14:57:40.81 <Info:KERN> Secondary Mgmt port
enabled
07/12/2007 14:57:40.80 <Info:KERN> Secondary Mgmt port
initialized
07/12/2007 14:57:34.70 <Info:SYST> Done initializing
MSM-B base module
07/12/2007 14:57:34.21 <Info:SYST> slave MSM
communications pipe is
open 
07/12/2007 14:57:33.64 <Info:SYST> MSM-B SN =
701021-00-05 0010F-07310
07/12/2007 14:57:33.53 <Info:SYST> Start initializing
MSM-B base module
07/12/2007 14:57:33.23 <Info:SYST> MSM-B is inserted.
07/12/2007 14:56:49.23 <Erro:SYST> MSM-B may have
problem to boot.
Reboot it.
07/12/2007 14:56:09.28 <Info:SYST> Removed MSM-B base
module
07/12/2007 14:56:09.28 <Info:KERN> Secondary Mgmt port
enabled
07/12/2007 14:56:09.27 <Info:SYST> slave MSM
communications pipe is
closed 
07/12/2007 14:56:05.31 <Info:SYST> slave MSM
communications pipe is
open 
07/12/2007 14:56:05.86 <Info:KERN> Secondary Mgmt port
initialized
07/12/2007 14:56:05.81 <Info:SYST> slave MSM
communications pipe is
closed 
07/12/2007 14:56:03.59 <Info:SYST> slave MSM
communications pipe is
open 
07/12/2007 14:56:03.55 <Crit:SYST> [2] Connection
between MSM-B mother
board port 31 and I/O module 8 port 7 broke, fix
immediately
07/12/2007 14:56:03.55 <Crit:SYST>
backplane=[701000-08 0219h00106 ]
07/12/2007 14:56:03.55 <Crit:SYST> slot
8=[701026-00-08 0034F-54284 ]
07/12/2007 14:56:03.55 <Crit:SYST> MSM-B=[701021-00-05
0010F-07310 ]
07/12/2007 14:56:03.55 <Crit:SYST> [2] Connection
between MSM-B mother
board port 30 and I/O module 7 port 7 broke, fix
immediately
07/12/2007 14:56:03.55 <Crit:SYST>
backplane=[701000-08 0219h00106 ]
07/12/2007 14:56:03.55 <Crit:SYST> slot
7=[701026-00-07 0029F-44881 ]
07/12/2007 14:56:03.55 <Crit:SYST> MSM-B=[701021-00-05
0010F-07310 ]
07/12/2007 14:56:03.55 <Crit:SYST> [2] Connection
between MSM-B mother
board port 29 and I/O module 6 port 7 broke, fix
immediately
07/12/2007 14:56:03.55 <Crit:SYST>
backplane=[701000-08 0219h00106 ]

I also read
http://www.extremenetworks.com/libr
aries/services/ErrorMessageDecoderRev4.pdf , but I
haven't found best solution for this kind of problem? Can
any one give me some advice?

Thank you in advance,
 Andraz


-- 
Humppa all the way!!

_______________________________________________
extreme-nsp mailing list
extreme-nsppuck.nether.net

https://puck.nether.net/mailman/listinfo/extreme-nsp

re

On Fri, 2007-07-13 at 10:38 +1200, TiM wrote:
> Based on your logs I'd say there's a good chance MSM-B
is broken
> (hardware failure).  Though having dealt with Extreme's
before, it
> could be any number of things really.

Is there any good change that I can replace MSM-B with a new
one
on-the-fly, without bringing system down. What's the
solution? Must be
in this case something typed into switch configuration?

I see that BD has put MSM-B on the side and it's not
recognized by the
system anymore.

MSM:              MSM-A        MSM-B        
                  ------------ ------------ 
Module Type:      MSM64i       Empty        
Current State:    MASTER       Empty  

> Your best bet is to open a TAC case, they'll be much
better at helping
> you.  

huh, don't know I we have any agreement of support with
Extreme or any
of there dealers.

> The only other advice I can offer is to run extended
diags on MSM-B,
> but in doing that there's always the chance it'll bring
the switch
> down.

I know :( It's a bit hard doing this on equipment that
running heavily
in productions ..

thanks,
 Andraz

-- 
Humppa all the way!!

_______________________________________________
extreme-nsp mailing list
extreme-nsppuck.nether.net

https://puck.nether.net/mailman/listinfo/extreme-nsp

--On 13 July 2007 11:08 +0200 Andraz Sraka <aaufbix.org> wrote:

> Is there any good change that I can replace MSM-B with
a new one
> on-the-fly, without bringing system down. What's the
solution? Must be
> in this case something typed into switch
configuration?

Yes. It's buggered. Hot swap of this MSM is supported by the
system.

Right now your running on one MSM and that not only means no
means of
failover, but 50% reduction in switching capacity.

It's helpful if the new MSM is running the same image as the
master MSM in
the system, but as I recall, not vital, but it's a while
since I've used
MSM64i hardware.

Mike
-- 
Mike Hughes     Chief Technical Officer  bond Internet
Exchange
mikelinx.net   http://www.linx.net/

_______________________________________________
extreme-nsp mailing list
extreme-nsppuck.nether.net

https://puck.nether.net/mailman/listinfo/extreme-nsp

> re
>
> On Fri, 2007-07-13 at 10:38 +1200, TiM wrote:
>> Based on your logs I'd say there's a good chance
MSM-B is broken
>> (hardware failure).  Though having dealt with
Extreme's before, it
>> could be any number of things really.
>
> Is there any good change that I can replace MSM-B with
a new one
> on-the-fly, without bringing system down. What's the
solution? Must be
> in this case something typed into switch
configuration?

Yes, there's a chance you can do that.  It should be as
simple as removing
the old one, inserting a new one then making sure it boots
up properly and
error free.
You could also just reseat MSM-B, in my experience this has
fixed more
than a few problems.

> I see that BD has put MSM-B on the side and it's not
recognized by the
> system anymore.
>
> MSM:              MSM-A        MSM-B
>                   ------------ ------------
> Module Type:      MSM64i       Empty
> Current State:    MASTER       Empty

An even better indication that MSM-B has failed outright
then.

>> Your best bet is to open a TAC case, they'll be
much better at helping
>> you.
>
> huh, don't know I we have any agreement of support with
Extreme or any
> of there dealers.

You can still log a case with Extreme, I think the cost is
$500 USD per
incident.  They will just suggest the same things I am
though, remove the
MSM, reinsert it, exchange it etc.  All carry a risk the BD
will crash and
need a reboot.

>> The only other advice I can offer is to run
extended diags on MSM-B,
>> but in doing that there's always the chance it'll
bring the switch
>> down.
>
> I know :( It's a bit hard doing this on equipment that
running heavily
> in productions ..

Of course, so I'd schedule an emergency outage on it for a
couple of
hours, get someone on the phone with some Extreme
experience, get a spare
MSM, cross your fingers and pray.

Just make sure you backup the config via TFTP first of all
before you do
anything.

Hope this helps you in some way.

Tim

_______________________________________________
extreme-nsp mailing list
extreme-nsppuck.nether.net

https://puck.nether.net/mailman/listinfo/extreme-nsp

Hi,

I would still be cautious (i.e. time it when it will least
impact your
users) when you replace this, as although as Mike says they
are
"officially" hot swap I have had a BD6808 reload
on me when replacing a
broken MSM-B (never had this problem with interface cards
though).

Cheers

Dan


> -----Original Message-----
> From: extreme-nsp-bouncespuck.nether.net 
> [mailto:extreme-nsp-bouncespuck.nether.net] On Behalf
Of Mike Hughes
> Sent: Friday, July 13, 2007 11:49 AM
> To: Andraz Sraka
> Cc: extreme-nsppuck.nether.net
> Subject: Re: [e-nsp] Critical errors: Connection
between 
> MSM-B mother board port X ..
> 
> --On 13 July 2007 11:08 +0200 Andraz Sraka <aaufbix.org> wrote:
> 
> > Is there any good change that I can replace MSM-B
with a new one 
> > on-the-fly, without bringing system down. What's
the 
> solution? Must be 
> > in this case something typed into switch
configuration?
> 
> Yes. It's buggered. Hot swap of this MSM is supported
by the system.
> 
> Right now your running on one MSM and that not only
means no 
> means of failover, but 50% reduction in switching
capacity.
> 
> It's helpful if the new MSM is running the same image
as the 
> master MSM in the system, but as I recall, not vital,
but 
> it's a while since I've used MSM64i hardware.
> 
> Mike
> -- 
> Mike Hughes     Chief Technical Officer  bond Internet
Exchange
> mikelinx.net   http://www.linx.net/
> 
> _______________________________________________
> extreme-nsp mailing list
> extreme-nsppuck.nether.net
> 
https://puck.nether.net/mailman/listinfo/extreme-nsp
> 
_______________________________________________
extreme-nsp mailing list
extreme-nsppuck.nether.net

https://puck.nether.net/mailman/listinfo/extreme-nsp

re

On Fri, 2007-07-13 at 11:49 +0100, Mike Hughes wrote:

> Yes. It's buggered. Hot swap of this MSM is supported
by the system.

And what are the changes if anything goes wrong. Because
I've been
testing hot-swapping on spare BD6808 and I successfully
fscked up two of
MSM cards :( One isn't booting anymore and the other has now
faulty
memory :( So much for hot-swapping. That's why I am not very
keen on
hot-swapping idea with equipment in production.

> Right now your running on one MSM and that not only
means no means of
> failover, but 50% reduction in switching capacity.

:(

regards,
 Andraz

-- 
Humppa all the way!!

_______________________________________________
extreme-nsp mailing list
extreme-nsppuck.nether.net

https://puck.nether.net/mailman/listinfo/extreme-nsp

re

On Fri, 2007-07-13 at 22:32 +1200, TiM wrote:

> Of course, so I'd schedule an emergency outage on it
for a couple of
> hours, get someone on the phone with some Extreme
experience, get a
> spare MSM, cross your fingers and pray.

I guess there will be a lot of praying going on 

> Just make sure you backup the config via TFTP first of
all before you do
> anything.

That's done on daily bases ..

thanks for the advice!

regards,
 Andraz

-- 
Humppa all the way!!

_______________________________________________
extreme-nsp mailing list
extreme-nsppuck.nether.net

https://puck.nether.net/mailman/listinfo/extreme-nsp