Freezing on Boot - system 75 message

[GuvnorIBrokeMyPCX]

Although this problem relates to a 6850 switch this question will be generic to AOS.

I have a switch which freezes on boot. It gets through 95% of the boot and then stops with a message ... (75) info message " switch logging device"

At this point console session freezes and the switches ports remain operationally down.
After about 3 minutes the console session shows some activity and then the switch then reboots........and the whole thing happens again.

During the little amount of console activity I see PMD files referenced (post mortem dumps).

Does anyone have any ideas what this could be. The switch is new out of box and worked initially. I loaded config on and then rebooted, this is when this happened. The same config was loaded onto 18 other switches without problem.

[GuvnorIBrokeMyPCX]

If no-one has any ideas can somebody let me know the process to hard-reset (factory-reset) the switch.

I have had a look at the options during boot interuption but the system isnt very intuitive and I cannot find docs on the net that may be of use to do stuff such as erase boot.cfg from the working and certified directories eg if you type erase you get info such as "erase [slot] (xxx<part>] [file name]" have tried multiple combinations of things but cant get anywhere.

Many Thanks in Advance

[cavagnaro]

If my memory doesn't fail me on those switches you had two config files:

current and factory, so you had to launch a restore command or delete the running config file and when reboot the OS restores the factory as the running one.

If wrong sorry but has been like a year without touching any switch...i believe there was a post about this procedure somewhere.

[benny]

Could you please attach the whole boot up process as a console log file? I would like to go through it.

Please note that the ID behind "SYSTEM" means nothing. 75 is just the ID for the SYSTEM application (0x4B) as TELNET would be 80 (0x50) etc., etc.

@cavagnaro:
You are right about the directories, but it is working and certified. If the two directories differ the switch will load from the certified directory unless it is told to boot from working (reload working no rollback-timeout).

Thanks,
Regards,
Benny

[GuvnorIBrokeMyPCX]

Thanks Benny.....as requested....

U-Boot 1.1.0(6.1.3.601.R01) (Sep 13 2006 - 14:37:19)

MPC8248 Reset Status: External Soft, External Hard

MPC8248 Clock Configuration
- Bus-to-Core Mult 4x, VCO Div 2, 60x Bus Freq 25-75 , Core Freq 100-300
- dfbrg 0, corecnf 0x1a, busdf 3, cpmdf 1, plldf 0, pllmf 3
- vco_out 400000000, scc_clk 100000000, brg_clk 100000000
- cpu_clk 400000000, cpm_clk 200000000, bus_clk 100000000
- pci_clk 66666666

CPU: MPC8248 (HiP7 Rev 14, Mask 1.0 1K50M) at 400 MHz
Board: Alcatel OS68xx (Kite II)
I2C: ready
DRAM: 256 MB
FLASH: 8 MB
In: serial
Out: serial
Err: serial

No ethernet found.
IDE: Bus 0: OK
Device 0: Model: CF 64MB Firm: 06/07/29 Ser#: GHS3ABPCF07052200004
Type: Removable Hard Disk
Capacity: 62.5 MB = 0.0 GB (128000 x 512)
USB: scanning bus for devices...
USB device not responding, giving up (status=80000000)
0 USB Devices found
Hit any key to stop autoboot: 1 0
## Booting image at ff900000 ...
Image Name: miniboot-6.1.3.601.R01
Image Type: PowerPC VxWorks Kernel Image (gzip compressed)
Data Size: 781716 Bytes = 763.4 kB
Load Address: 00005000
Entry Point: 00005000
Verifying Checksum ... OK
Uncompressing Kernel Image ... OK
## Ethernet MAC address not copied to NV RAM
Loading .text @ 0x01000000 (1203372 bytes)
Loading .rodata @ 0x01125cb0 (242483 bytes)
Loading .data @ 0x01160fe8 (109988 bytes)
Clearing .sbss @ 0x0117bd90 (1556 bytes)
Clearing .bss @ 0x0117d000 (1818940 bytes)
## Using bootline (@ 0x4200): flash(1,0)cmm:default h=192.168.10.1 e=192.168.10.1:ffffff00 cb=9600 cp=none cs=1 cw=8 cm=modemControlOff f=0
## Starting vxWorks at 0x01000000 ...
ataDrv call. 0 1 24 24 621 0 0

CS EEPROM Module ID: 42012219 , System type: 48 port Lite 100Base-X, with 4 combo ports, stacking- kite2


Adding 4804 symbols for standalone.


Boot parameters are:

'flash(1,0)cmm:default e=10.255.24.81:ff000000 h=192.168.10.1 g=10.255.24.181 cb=9600 cp=none cs=1 cw=8 cm=modemControlOff'


AOS miniboot version: 6.1.3.601.R01
VxWorks (for Alcatel CMM MPC8248) version VxWorks5.5.1
KERNEL: WIND version 2.6
Made on Sep 13 2006, 14:39:14.
Copyright Wind River Systems, Inc., 1984-2006

CPU: Alcatel CMM MPC8248. Processor #0.
Memory Size: 0xfa00000 (250 MB). BSP version 1.2/1.

Portions of code copyrighted by Agranat Systems, Inc. All Rights reserved
Portions of code copyrighted by Alcatel Internetworking, Inc. 1994-2006. All Rights reserved
Portions of code copyrighted by Apptitude, Inc. 1997-1999. All Rights reserved
Portions of code copyrighted by Free Software Foundation, Inc. 1989,1991. All rights reserved
Portions of code copyrighted by The OpenLDAP Foundation 1999-2000. All rights reserved
Portions of code copyrighted by Theodore T'so 1994,1995,1996,1997,1998,1999. All Rights reserved
Portions of code copyrighted by RSA Security, Inc. All Rights reserved
Portions of code copyrighted by Sun Microsystems, Inc. All Rights reserved
Portions of code copyrighted by Wind River Systems, Inc. All Rights reserved
Portions of code developed by the University of California, Berkeley.
Portions of code developed by the Carnegie-Mellon University.


Attempting to load from the CERTIFIED directory...
Loading kernel from '/flash/certified/K2os.img'...
5365776
Loaded 'alcatelNetworking_kernel.lnk' from '/flash/certified/K2os.img'.
Starting at 0x100000...

ataDrv call. 0 1 24 24 621 0 0

Adding 9877 symbols for standalone.


Boot parameters are:

'flash(1,0)cmm:default e=10.255.24.81:ff000000 h=192.168.10.1 g=10.255.24.181 cb=9600 cp=none cs=1 cw=8 cm=modemControlOff'


AOS kernel version: 6.1.3.719.R01
VxWorks (for Alcatel CMM MPC8248) version VxWorks5.5.1
KERNEL: WIND version 2.6
Made on Dec 6 2006, 18:54:49.
Copyright Wind River Systems, Inc., 1984-2006

CPU: Alcatel CMM MPC8248. Processor #0.
Memory Size: 0xfa00000 (250 MB). BSP version 1.2/1.

Portions of code copyrighted by Agranat Systems, Inc. All Rights reserved
Portions of code copyrighted by Alcatel Internetworking, Inc. 1994-2006. All Rights reserved
Portions of code copyrighted by Apptitude, Inc. 1997-1999. All Rights reserved
Portions of code copyrighted by Free Software Foundation, Inc. 1989,1991. All rights reserved
Portions of code copyrighted by The OpenLDAP Foundation 1999-2000. All rights reserved
Portions of code copyrighted by Theodore T'so 1994,1995,1996,1997,1998,1999. All Rights reserved
Portions of code copyrighted by RSA Security, Inc. All Rights reserved
Portions of code copyrighted by Sun Microsystems, Inc. All Rights reserved
Portions of code copyrighted by Wind River Systems, Inc. All Rights reserved
Portions of code developed by the University of California, Berkeley.
Portions of code developed by the Carnegie-Mellon University.


0xf9ffde8 (tRootTask): cmm_random_init OK
0xefb44b0 (tWhirlpool): batch_entropy_process called
Target Name: vxTarget
User: target
Attaching network interface lo0... done.

The current date and time is:
Coordinated Universal Time (UTC): TUE NOV 06 09:36:02 2007
Local time: TUE NOV 06 09:36:02 2007 GMT

Starting dynamic software load from '/flash/certified/'...
Creating archive cache...
Boot as 10/100 FE.

TUE NOV 06 09:36:15 : SYSTEM (75) info message:
+++ Switch Logging device '/dev/console' enabled for output

TUE NOV 06 09:36:15 : CSM-CHASSIS (103) info message:
+++ == CSM == === Chassis Supervision Starting Up ===
PCI Discovery: found unit 0: BCM56502_B2 on bus 0, dev 0x10, fn 0
Using driver: BCM56504_B0
PCI unit 0: Dev 0xb502, Rev 0x13, Chip BCM56502_B2, Driver BCM56504_B0
PCI Discovery: found unit 1: BCM56504_B2 on bus 0, dev 0x11, fn 0
Using driver: BCM56504_B0
PCI unit 1: Dev 0xb504, Rev 0x13, Chip BCM56504_B2, Driver BCM56504_B0

KiteII System Initialized.
sysStartWatchdog() Starting watchdog timer to INTERRUPT after 60 secs.

TUE NOV 06 09:36:34 : CSM-CHASSIS (103) info message:
+++ == CSM == APP MONITORING ON
+++ == CSM == ===== THIS STACK IS PRIMARY =====
Boot as 10/100 FE.

TUE NOV 06 09:36:38 : HSM-CHASSIS (101) info message:
+++ T1:Ni Insertion detected on slot 8
+++ == HSM == Power Supply 22 Inserted
+++ ==== HSM === Power Supply 22 is OPERATIONAL

TUE NOV 06 09:36:59 : WEB (69) info message:
+++ WebView SSL/RSA file-based certificate set-up successful!

TUE NOV 06 09:37:05 : SYSTEM (75) info message:
+++ Switch Logging device '172.16.80.97' enabled for output

There is then a 5 minute gap and the following output is generated...


TUE NOV 06 09:42:24 : CSM-CHASSIS (103) error message:
+++ ********* CS Debug: STR FATAL **********
+++ Source file name : cs_csm_fsm_action.c


Dumping PMD.....


tCsCSMtask (dd4d168) @ 1 READY lckd=0 EE ME DS stk dd4d168-dd45c38
(dd4d108)
2e0e1c vxTaskEntry +5c: csCsmMain()
(dd4cc90)
ebf340 csCsmMain +a68: csCsmFsm()
(dd4cc60)
e9b89c csCsmFsm +24: csCsmSubFsm()
(dd4cc10)
e9944c csCsmSubFsm +224: CSM_P1_4()
(dd4cbc0)
e9d6ec CSM_P1_4 +51c: csCsmSendCsmStr()
(dd4cb98)
e9739c csCsmSendCsmStr +90: csCsmStackRebootThisCmm()
(dd4cb80)
eba494 csCsmStackRebootThisCmm +144: pmdDump()
(dd4cb58)
133970 pmdDump +74: _tt()
+++ Line number in the file: 1961
Dumping workQ

Many Thanks

[benny]

Hi,

I am sorry - It does not ring any bell. Never saw that.

Method 1.) You can factory reset the 6850 using the "u-boot" commands.

*** Stop the Boot when prompted *** (Hit any key to stop autoboot: 0)
=> fatls ide :1,0 <- that gives you the "ls output" of the /flash system
=> fatdelete ide :1,0 /working/boot.cfg
=> fatdelete ide :1,0 /certified/boot.cfg

Method 2.) In case that you prefer to work with the miniboot you have to do the following.
*** Stop the Boot when prompted *** (Hit any key to stop autoboot: 0)
=> printenv (that you know the default settings)
=> setenv bootflags 0x1000
=> printenv (note that the bootflags were updated)
=> saveenv
=> run miniboot

In miniboot it is the usual command set (if I remember correctly) xdelete "boot.cfg" (of course you have to change in the working/certified directory first...)

Type the command "reboot" in miniboot to reboot the switch and change the bootflags in uboot back to 0x0.

Of course this is no guarantee that your switch is working again afterwards.

Please let me know the result ...

Regards,
Benny

[GuvnorIBrokeMyPCX]

Benny, Thanks a lot.

I used method 1 which has solved the problem.

It was obviously an issue with the boot.cfg file. Must have got corrupted when I transfered it via console initially. To be honest I usually prep the file and transfer to switch via FTP and then reload the config......done it 100s of times without a prob. I will avoid text transfer via console in future as 2 out of 20 switches failed and ended up with currupt boot.cfg files.

Thanks Again.

[benny]

I had always problems with pasting configuration using SecureCRT. The config got messed up - lines were lost - regardless if it was Foundry, Alcatel(-Lucent) or Cisco.

I found that there was a patch for SecureCRT to fix that but since then I simply avoid pasting huge parts of a configuration.

I am happy that your switches work again.

Regards,
Benny