Hello Guys,
Just trying to see if anyone has a better understanding
about those logs
than me. Basically one of my major routers "M7i"
failed last week ,
seems that the Forward Engine stops and brought all physical
interfaces
down on the router, causing full loss of traffic, I have
Junos 7.4 R2.6
running on all my M7i routers across the network. Below I
have a few
logs from the crash time. Basically after a minute the CFEB
did reboot
by itself and got the routing capability back. The router it
self didn't
reboot like I saw system uptime 200 days, only the CFEB did
a reboot.
In case anyone wants to share any thought I will
appreciate.
Thanks
Rommel
Mar 28 09:43:11 Guinness /kernel: rdp keepalive expired,
connection
dropped - src 1:1021 dest 2:11264
Mar 28 09:43:11 Guinness craftd[2684]: Major alarm set,
CFEB not
online, the box is not forwarding
Mar 28 09:43:11 Guinness chassisd[2682]:
CHASSISD_SHUTDOWN_NOTICE:
Shutdown reason: CFEB connection lost
Mar 28 09:43:11 Guinness alarmd[2683]: Alarm set: CFEB
color=RED,
class=CHASSIS, reason=CFEB not online, the box is not
forwarding
Mar 28 09:43:11 Guinness craftd[2684]: forwarding display
request to
chassisd: type = 4, subtype = 43
Mar 28 09:43:11 Guinness chassisd[2682]:
CHASSISD_IFDEV_DETACH_FPC:
ifdev_detach(0)
Mar 28 09:43:11 Guinness /kernel: aggr_link_op: link
so-0/0/0.0
(lidx=0) detached from bundle as0.0
Mar 28 09:43:11 Guinness syslogd: sendto: Network is down
Mar 28 09:43:11 Guinness /kernel: so-0/0/0.0 leaves as0.0
Mar 28 09:43:11 Guinness /kernel: cos_msg_sched_policy_def:
ifd not
found (idx 0x88)
Mar 28 09:43:11 Guinness /kernel: bundle as0: bundle IFD
state changed
to DOWN
Mar 28 09:43:11 Guinness /kernel: aggr_link_op: link
so-0/0/1.0
(lidx=1) detached from bundle as0.0
Mar 28 09:43:11 Guinness /kernel: bundle as0.0: bundle IFL
state
changed to DOWN
Mar 28 09:43:11 Guinness /kernel: so-0/0/1.0 leaves as0.0
Mar 28 09:43:11 Guinness /kernel: cos_msg_sched_policy_def:
ifd not
found (idx 0x89)
Mar 28 09:43:11 Guinness snmpd[2687]: SNMPD_SEND_FAILURE:
trap_io_send_trap_now: send to (87.192.235.219) failure:
Network is down
Mar 28 09:43:11 Guinness chassisd[2682]:
CHASSISD_IFDEV_DETACH_FPC:
ifdev_detach(1)
Mar 28 09:43:11 Guinness snmpd[2687]: SNMPD_TRAP_QUEUED:
Adding trap to
87.192.235.219 to destination queue, 1 traps in queue
Mar 28 09:43:11 Guinness snmpd[2687]: SNMPD_SEND_FAILURE:
trap_io_send_trap_now: send to (62.231.52.94) failure:
Network is down
Mar 28 09:43:11 Guinness /kernel: bundle as1: bundle IFD
state changed
to DOWN
Mar 28 09:43:11 Guinness /kernel: aggr_link_op: link
so-0/0/2.0
(lidx=0) detached from bundle as1.0
Mar 28 09:43:11 Guinness /kernel: bundle as1.0: bundle IFL
state
changed to DOWN
Mar 28 09:43:11 Guinness /kernel: so-0/0/2.0 leaves as1.0
Mar 28 09:43:11 Guinness /kernel: cos_msg_sched_policy_def:
ifd not
found (idx 0x8a)
Mar 28 09:43:11 Guinness snmpd[2687]: SNMPD_TRAP_QUEUED:
Adding trap to
62.231.52.94 to destination queue, 1 traps in queue
Mar 28 09:43:11 Guinness snmpd[2687]: SNMPD_SEND_FAILURE:
trap_io_send_trap_now: send to (10.10.1.43) failure: Network
is down
Mar 28 09:43:12 Guinness snmpd[2687]: SNMPD_TRAP_QUEUED:
Adding trap to
10.10.1.43 to destination queue, 1 traps in queue
Mar 28 09:43:12 Guinness snmpd[2687]: SNMPD_TRAP_QUEUED:
Adding trap to
87.192.235.219 to destination queue, 2 traps in queue
Mar 28 09:43:12 Guinness /kernel: aggr_link_op: link
so-0/0/3.0
(lidx=1) detached from bundle as1.0
Mar 28 09:43:12 Guinness /kernel: so-0/0/3.0 leaves as1.0
Mar 28 09:46:38 Guinness snmpd[2687]: SNMPD_TRAP_QUEUED:
Adding trap to
10.10.1.43 to destination queue, 266 traps in queue
Mar 28 09:46:38 Guinness /kernel: fxp1: media DOWN 100Mb /
full-duplex
Mar 28 09:46:39 Guinness /kernel: fxp1: link UP 100Mb /
full-duplex
Mar 28 09:46:39 Guinness snmpd[2687]: SNMPD_TRAP_QUEUED:
Adding trap to
87.192.235.219 to destination queue, 267 traps in queue
Mar 28 09:46:39 Guinness mib2d[2688]: SNMP_TRAP_LINK_UP:
ifIndex 2,
ifAdminStatus up(1), ifOperStatus up(1), ifName fxp1
Mar 28 09:46:39 Guinness snmpd[2687]: SNMPD_TRAP_QUEUED:
Adding trap to
62.231.52.94 to destination queue, 267 traps in queue
Mar 28 09:46:39 Guinness snmpd[2687]: SNMPD_TRAP_QUEUED:
Adding trap to
10.10.1.43 to destination queue, 267 traps in queue
Mar 28 09:46:39 Guinness snmpd[2687]: SNMPD_TRAP_QUEUED:
Adding trap to
87.192.235.219 to destination queue, 268 traps in queue
Mar 28 09:46:39 Guinness snmpd[2687]: SNMPD_TRAP_QUEUED:
Adding trap to
62.231.52.94 to destination queue, 268 traps in queue
Mar 28 09:46:39 Guinness snmpd[2687]: SNMPD_TRAP_QUEUED:
Adding trap to
10.10.1.43 to destination queue, 268 traps in queue
Mar 28 09:46:39 Guinness cfeb COSMAN PFE RED prof_id_start
1000,
prof_id_end 3048
Mar 28 09:46:39 Guinness cfeb CSBR: Reset reason (0x8): RE
Initiated
Reset
Mar 28 09:46:39 Guinness cfeb SNTP: Daemon created
Mar 28 09:46:39 Guinness cfeb Version 7.4R2.6 by builder on
2006-01-20
14:09:28 UTC
Mar 28 09:46:39 Guinness cfeb On-board NVRAM contains
diagnostic
information.
Mar 28 09:46:39 Guinness cfeb CCHIP: Detected CF 2.0
Mar 28 09:46:39 Guinness cfeb SNTPD: Initial time of day
set.
Mar 28 09:46:42 Guinness snmpd[2687]: SNMPD_SEND_FAILURE:
trap_io_send_trap_now: send to (10.10.1.43) failure: No
route to host
Mar 28 09:46:42 Guinness snmpd[2687]: SNMPD_SEND_FAILURE:
trap_io_send_trap_now: send to (62.231.52.94) failure: No
route to host
Mar 28 09:46:42 Guinness snmpd[2687]: SNMPD_SEND_FAILURE:
trap_io_send_trap_now: send to (87.192.235.219) failure: No
route to
host
Mar 28 09:46:47 Guinness cfeb CM: Established connection to
routing
engine
Mar 28 09:46:47 Guinness /kernel: pfe_peer_alloc: type 1,
index 0, old
state Closed new state Opening mastership 1
Mar 28 09:46:47 Guinness /kernel: pfe_peer_set_timeout: set
timeout
0xc03990a9
Mar 28 09:46:47 Guinness /kernel: pfe_listener_connect:
conn
established: listener idx=0, tnpaddr=0x2
Mar 28 09:46:47 Guinness chassisd[2682]:
CHASSISD_SNMP_TRAP10: SNMP
trap generated: FRU power on (jnxFruContentsIndex 6,
jnxFruL1Index 1,
jnxFruL2Index 0, jnxFruL3Index 0, jnxFruName CFEB,
jnxFruType 4,
jnxFruSlot 1, jnxFruOfflineReason 2, jnxFruLastPowerOff
60104671,
jnxFruLastPowerOn 60104671)
Mar 28 09:46:47 Guinness chassisd[2682]:
CHASSISD_TIMER_CLR_ERR:
timer_clear_fru: unable to clear timeout
Mar 28 09:46:47 Guinness snmpd[2687]: SNMPD_TRAP_QUEUED:
Adding trap to
87.192.235.219 to destination queue, 269 traps in queue
Mar 28 09:46:47 Guinness chassisd[2682]:
CHASSISD_IFDEV_DETACH_ALL_PSEUDO: ifdev_detach(pseudo
devices: all)
Mar 28 09:46:47 Guinness snmpd[2687]: SNMPD_TRAP_QUEUED:
Adding trap to
62.231.52.94 to destination queue, 269 traps in queue
Mar 28 09:46:47 Guinness cfeb PFEMAN: Session manager
active
Mar 28 09:46:47 Guinness cfeb PFEMAN: Established
connection to Master
Mar 28 09:46:47 Guinness cfeb CM: ALARM SET: (Major) Slot
0: CFEB not
online, the box is not forwarding
Mar 28 09:46:47 Guinness snmpd[2687]: SNMPD_TRAP_QUEUED:
Adding trap to
10.10.1.43 to destination queue, 269 traps in queue
Mar 28 09:46:47 Guinness chassisd[2682]:
CHASSISD_IFDEV_CREATE_NOTICE:
create_pseudos: created pseudo interface device for as0
Mar 28 09:46:47 Guinness snmpd[2687]: SNMPD_TRAP_QUEUED:
Adding trap to
87.192.235.219 to destination queue, 270 traps in queue
Mar 28 09:46:47 Guinness snmpd[2687]: SNMPD_TRAP_QUEUED:
Adding trap to
62.231.52.94 to destination queue, 270 traps in queue
Mar 28 09:46:47 Guinness chassisd[2682]:
CHASSISD_IFDEV_CREATE_NOTICE:
create_pseudos: created pseudo interface device for as1
Mar 28 09:46:47 Guinness snmpd[2687]: SNMPD_TRAP_QUEUED:
Adding trap to
10.10.1.43 to destination queue, 270 traps in queue
Mar 28 09:46:47 Guinness chassisd[2682]:
CHASSISD_IFDEV_CREATE_NOTICE:
create_pseudos: created pseudo interface device for as2
Mar 28 09:46:47 Guinness chassisd[2682]:
CHASSISD_IFDEV_CREATE_NOTICE:
create_pseudos: created pseudo interface device for as3
Mar 28 09:46:47 Guinness chassisd[2682]:
CHASSISD_IFDEV_CREATE_NOTICE:
create_pseudos: created pseudo interface device for as4
Mar 28 09:46:47 Guinness cfeb CM: ALARM SET: (Major) Slot
0: Host 0
fxp0: Ethernet Link Down
Mar 28 09:46:47 Guinness chassisd[2682]:
CHASSISD_IFDEV_CREATE_NOTICE:
create_pseudos: created pseudo interface device for as5
Mar 28 09:46:47 Guinness chassisd[2682]:
CHASSISD_IFDEV_CREATE_NOTICE:
create_pseudos: created pseudo interface device for as6
Mar 28 09:46:47 Guinness chassisd[2682]:
CHASSISD_IFDEV_CREATE_NOTICE:
create_pseudos: created pseudo interface device for as7
Mar 28 09:46:49 Guinness chassisd[2682]:
CHASSISD_FRU_EVENT:
fpc_m40_recv_restart: restarted FPC 0
Mar 28 09:46:49 Guinness chassisd[2682]:
CHASSISD_FRU_EVENT:
fpc_m40_recv_restart: restarted FPC 1
Mar 28 09:46:57 Guinness craftd[2684]: Major alarm cleared,
CFEB not
online, the box is not forwarding
Mar 28 09:46:57 Guinness alarmd[2683]: Alarm cleared: CFEB
color=RED,
class=CHASSIS, reason=CFEB not online, the box is not
forwarding
Mar 28 09:46:57 Guinness craftd[2684]: forwarding display
request to
chassisd: type = 4, subtype = 44
Mar 28 09:46:59 Guinness cfeb CM: ALARM CLEAR: Slot 0: CFEB
not online,
the box is not forwarding
Got back up after reset the Cfeb
Mar 28 09:47:36 Guinness snmpd[2687]: SNMPD_TRAP_QUEUED:
Adding trap to
10.10.1.43 to destination queue, 342 traps in queue
Mar 28 09:47:36 Guinness /kernel: bundle as0: bundle IFD
state changed
to UP
Mar 28 09:47:36 Guinness /kernel: bundle as1: bundle IFD
state changed
to UP
Note:
This message is for the named person's use only. It may
contain confidential, proprietary or legally privileged
information. No confidentiality or privilege is waived or
lost by any mistransmission. If you receive this message in
error, please immediately delete it and all copies of it
from your system, destroy any hard copies of it and notify
the sender. You must not, directly or indirectly, use,
disclose, distribute, print, or copy any part of this
message if you are not the intended recipient. Irish
Broadband and any of its subsidiaries each reserve the right
to monitor all e-mail communications through its networks.
Any views expressed in this message are those of the
individual sender, except where the message states otherwise
and the sender is authorized to state them to be the views
of any such entity.
Thank You.
_______________________________________________
juniper-nsp mailing list juniper-nsp puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp
|
