Won't boot to Certified

[justinf4]

I've got a 2 OS6450-48, 1 OS6450-P48, and 4 OS6450-P10 switches that are doing this.

AOS BootSelector Version: 6.6.3.259.R01

Creation Date: Mar 21 2012 23:07:59.

Press x to choose XMODEM...

Press s to STOP AT MINIBOOT...

Reading kfminiboot.bs from /boot ...done:
1452091

Uncompressed kfminiboot.bs @ 0xA10000...
Running Miniboot...

Detected board DB-98DX4122-48G
Ctrl env init... Done.
CPU if init..... Done.
Done. Actual freq = 96153Hz
Memory on board initialized .

Init Phy port 0 .. Done

Init Phy port 1 .. Done
Power up GMII port0 .. Done.
Power up GMII port1 .. Done.


Detected DRAM size 256MB
Detect SysClk 320MHz
Detect Tclk 200MHz
Marvell Controller initialization Done.
Eth int Ctrl.... Done
GPIO int Ctrl... Done
PEX int Ctrl.... Done
PEX0 init....... Done. DEVICE type selected
USB Init........ Done
USB int Ctrl.... Done
XOR int Ctrl.... Done
XOR init........ Done
mvCtrlInit2..... done
PCI auto config. done
sysHwInit2...... DoneUSB2 Host Stack Initialized.
USB Hub Driver Initialized
USBD Wind River Systems, Inc. 560 Initialized
Non-PCI USB device found: pgmIf=0xa0, idx=0 , baseAddr=0xf1050100, irq=19, swapF
unc=0xa1abf4
EHCI Controller found.
Waiting to attach to USBD...Done.
0xffffe50 (tRootTask): usbBulkDevInit() returned OK
usbCbiUfiDevInit() returned OK
CBI Class Driver Successfully Initialized

CS EEPROM Module ID: 420122b2 , System type: 6450 8 POE ports, 2 combo ports & 2
fixed sfp stacking or uplink port - Kingfisher2


System Configuration

BSP Ver: 1.2/1.1.01 Creation date: Mar 22 2012, 00:07:53
CHIP : MV88F6281 Rev 2
CPU : ARM926 (Rev 1)
BOARD : DB-98DX4122-48G (0x7)
CLOCKs : tClk = 200 MHz, sysClk = 320 MHz
pClk = 800 MHz

NAT_SUPPORT = Disabled
VFP_SUPPORT = Disabled
FlashFX initialization required 799ms
File System initialization required 0ms
FlashFX initialization required 99ms
File System initialization required 0ms
Target Name: vxTarget
Ethernet port #0 initialized
mgi0 Interface Loaded
mgi0 Interface Started
mgi0 Interface - Link Up, 1000 Full
Attaching network interface lo0... done.
mgi0 Interface - Link Up, 1000 Full

Adding 8031 symbols for standalone.


Boot parameters are:

'flash(0,0)cmm:default e=10.4.50.165:ff000000 g=10.4.50.5 u=admin pw=switch cb=
9600 cp=none cs=1 cw=8 cm=modemControlOff'


AOS miniboot version: 6.6.3.259.R01
VxWorks (for DB-98DX4122-48G BE MMU ARCH 5) version VxWorks5.5.1
KERNEL: WIND version 2.6
Made on Mar 22 2012, 00:07:53.
Copyright Wind River Systems, Inc., 1984-2006

CPU: DB-98DX4122-48G BE MMU ARCH 5. Processor #0.
Memory Size: 256 MBytes. BSP version 1.2/1.1.01.

Portions of code copyrighted by Agranat Systems, Inc. All Rights reserved
Portions of code copyrighted by Alcatel Internetworking, Inc. 1994-2012. All Ri
ghts reserved
Portions of code copyrighted by Apptitude, Inc. 1997-1999. All Rights reserved
Portions of code copyrighted by Free Software Foundation, Inc. 1989,1991. All r
ights reserved
Portions of code copyrighted by The OpenLDAP Foundation 1999-2000. All rights r
eserved
Portions of code copyrighted by Theodore T'so 1994,1995,1996,1997,1998,1999. Al
l Rights reserved
Portions of code copyrighted by RSA Security, Inc. All Rights reserved
Portions of code copyrighted by Sun Microsystems, Inc. All Rights reserved
Portions of code copyrighted by Wind River Systems, Inc. All Rights reserved
Portions of code developed by the University of California, Berkeley.
Portions of code developed by the Carnegie-Mellon University.


Transferring crash data to a file ...

length of pmd data 0xdabe6
Transferring PMD done...
Attempting to load from the CERTIFIED directory...
Loading kernel from '/flash/certified/KFos.img'...
7211712
Loade
Detected board DB-98DX4122-48G
Ctrl env init... Done.
CPU if init..... Done.
Done. Actual freq = 96153Hz
Memory on board initialized .

Init Phy port 0 .. Done

Init Phy port 1 .. Done
Power up GMII port0 .. Done.
Power up GMII port1 .. Done.


Detected DRAM size 256MB
Detect SysClk 320MHz
Detect Tclk 200MHz
Marvell Controller initialization Done.
Eth int Ctrl.... Done
GPIO int Ctrl... Done
PEX int Ctrl.... Done
PEX0 init....... Done. DEVICE type selected
USB Init........ Done
USB int Ctrl.... Done
XOR int Ctrl.... Done
XOR init........ Done
mvCtrlInit2..... done
PCI auto config. done
sysHwInit2...... DoneUSB2 Host Stack Initialized.
USB Hub Driver Initialized
USBD Wind River Systems, Inc. 560 Initialized
Non-PCI USB device found: pgmIf=0xa0, idx=0 , baseAddr=0xf1050100, irq=19, swapF
unc=0x10abf4
EHCI Controller found.
Waiting to attach to USBD...Done.
0xffffe50 (tRootTask): usbBulkDevInit() returned OK
usbCbiUfiDevInit() returned OK
CBI Class Driver Successfully Initialized

System Configuration

BSP Ver: 1.2/1.1.01 Creation date: May 14 2012, 13:15:10
CHIP : MV88F6281 Rev 2
CPU : ARM926 (Rev 1)
BOARD : DB-98DX4122-48G (0x7)
CLOCKs : tClk = 200 MHz, sysClk = 320 MHz
pClk = 800 MHz

NAT_SUPPORT = Disabled
VFP_SUPPORT = Disabled
FlashFX initialization required 799ms
File System initialization required 0ms
FlashFX initialization required 99ms
File System initialization required 0ms
Target Name: vxTarget
Attaching network interface lo0... done.
Write Protecting exception Vectors

Write protecting KFos .text sections (0x100000 to 0x523000) 1059 pages

Adding 11926 symbols for standalone.


Boot parameters are:

'flash(0,0)cmm:default e=10.4.50.165:ff000000 g=10.4.50.5 u=admin pw=switch cb=
9600 cp=none cs=1 cw=8 cm=modemControlOff'


AOS kernel version: 6.6.3.372.R01
VxWorks (for DB-98DX4122-48G BE MMU ARCH 5) version VxWorks5.5.1
KERNEL: WIND version 2.6
Made on May 14 2012, 13:15:10.
Copyright Wind River Systems, Inc., 1984-2006

CPU: DB-98DX4122-48G BE MMU ARCH 5. Processor #0.
Memory Size: 256 MBytes. BSP version 1.2/1.1.01.

Portions of code copyrighted by Agranat Systems, Inc. All Rights reserved
Portions of code copyrighted by Alcatel Internetworking, Inc. 1994-2012. All Ri
ghts reserved
Portions of code copyrighted by Apptitude, Inc. 1997-1999. All Rights reserved
Portions of code copyrighted by Free Software Foundation, Inc. 1989,1991. All r
ights reserved
Portions of code copyrighted by The OpenLDAP Foundation 1999-2000. All rights r
eserved
Portions of code copyrighted by Theodore T'so 1994,1995,1996,1997,1998,1999. Al
l Rights reserved
Portions of code copyrighted by RSA Security, Inc. All Rights reserved
Portions of code copyrighted by Sun Microsystems, Inc. All Rights reserved
Portions of code copyrighted by Wind River Systems, Inc. All Rights reserved
Portions of code developed by the University of California, Berkeley.
Portions of code developed by the Carnegie-Mellon University.



The current date and time is:
Coordinated Universal Time (UTC): SAT FEB 03 18:01:42 2001
Local time: SAT FEB 03 18:01:42 2001 UTC

Starting dynamic software load from '/flash/certified/'...
Unable to load image file 'loadBase.lnk'.
Certified: [Kernel]->

If I run the command "setNextRunningVersion 2" and "reboot" it will reboot into working.
When I run a "ls" in the certified directory, KFBase.img has a ? as the file size.

Any help would be helpful!

[devnull]

Never had this with my 6450s.

What is with the kbase in working?
Can you issue copy working certified? Does that fix the "?"
Will it boot afterwards?

[justinf4]

Can't copy working to certified. It errors out trying to copy KFBase.img. Five to ten minutes pass and then all the files in the certified directory show up with ? where their file size should be. A few minutes after that, the files in working do the same and a few minutes after that the entire /flash file system disappears. The switch then does a pmd dump and tries to reboot to certified and back to square 1.

[devnull]

Any chance to upload most recent code (6.6.3.413) to working? e.g. ftping it there?
reload working no rollback timeout afterwards

[one6f]

Try to check your flash with
-> fsck /flash
or from dshell
Certified: [Kernel]->dosFsShow ("/flash",1)

Here is AOS recovery and Zmodem procedure

[justinf4]

Tried upgrading the code, problem persisted.
Ran FSCK /FLASH and it came up with the following errors:

Error found in inode tree of the following:
/certified/KFbase.img
Results of check on all directory contents and inodes.........ERROR

Errors Detected:
Invalid Inode Meta Block Encountered.

Volume Information:
2,048 bytes per block.
55,104 total blocks in volume.
relFsChk Error = 209.

Error (209) checking the media
An error was encountered during fsck processing.
This error was not repaired by fsck.
ERROR: Call to fsck failed.
ERROR: resource allocation failure

Tried formatting the flash and reloading and the problem persists.
Alcatel has determined a bad flash component in the switches and has issued an RMA.
This has happened with 8 switches so far. All switches were ordered at approximately the same time (May 2012) and have close serial numbers.

[one6f]

Too bad
And who was the manufacturer of this flash memory?

[benny]

I suggest you open a SR with Alcatel-Lucent, that fsck can't repair the issues is a bad sign.

Benny

[wilfie113]

I had the exact same issue but was able to resolve the issue on one switch. Here is what I did:
1. Boot switch in Miniboot
2. Issued SetNextRunningVersion 2 to boot switch in working memory
4. Issued newfs /flash command to format flash memory
5. Made new /flash/certified and /flash/working directories
6. Copied files from /uflash to /flash/certified directory
7. Issued reload command to reboot switch
8. Issued copy certified working command
9. Ran a fsck but found no errors afterwards

[stewartd007]

I have experienced exactly the same issues on a 6450-P24 and a 6450-48. I managed to make one work again by carrying out a similar process to what wilfie113 did, although I had booted from USB to get files back onto the switch. The switch now appears to work but I have not been brave enough to put back on customer site yet in case it fails again. I have not attempted the same on my 2nd faulty switch yet as I was wanting Alcatel to identify the cause of the issue - hardware fault or software fault.

Alcatel have only asked me to send tech support logs and apparently do not know of this issue. Is there a range of Serial numbers affected by this fault? Are Alcatel offering replacement switches for ones experiencing the problems?

I currently have 8 x 6450 switches, the first 2 that I installed are the ones that failed. I currently have several other 6450 switches on customer sites - should I be worried about these too?

Will keep you posted of any progress!