Omniswitch 6850 lost chains in flash

[orochi_y]

Hi Benny & experts,

One of my Omniswitch is having the below issue:

-> fsck /flash
/flash/ - disk check in progress ...
"/flash - lost chains " has illegal cluster in chain
/flash/ - Errors detected. To correct disk structure, please start check disk with a permission to repair.

total # of clusters: 62,317
# of free clusters: 15,961
# of bad clusters: 0
total free space: 16,344,064
max contiguous free space: 8,500,224 bytes
# of files: 59
# of folders: 5
total bytes in files: 47,401,530
# of lost chains: 1
total bytes in lost chains: 24,576
An error was encountered during fsck processing.
This error was not repaired by fsck.
ERROR: Call to fsck failed.
ERROR: resource allocation failure


I tried using the command fsck /flash repair, but got no luck because the AOS doesn't have this command available. The version of AOS as below:

System:
Description: 6.3.1.871.R01 GA, December 20, 2007.,
Object ID: 1.3.6.1.4.1.6486.800.1.1.2.1.7.1.11,
Up Time: 0 days 2 hours 41 minutes and 13 seconds,
Contact: Alcatel-Lucent, http://alcatel-lucent.com/wps/portal/enterprise,
Name: 03F-LANRM-ES01,
Location: Level 3 Office Network,
Services: 72,
Date & Time: MON AUG 03 2009 12:37:46 (LOCAL)

Flash Space:
Primary CMM:
Available (bytes): 16345088,
Comments : Recommend Cleanup to 23428992 bytes free


Currently the switch will always hang without notice.

Thanks
Orochi

[benny]

Hi Orochi and welcome to the board,

It is correct that the 6.3.1.R01 GA had an issue with the FSCK function call, this is resolved in later 6.3.1.R01 maintenance releases.

To resolve your issue ...

We need to format the flash to make everything clean - but there are some action items prior to do so.

I assume you run a standalone switch which is not stacked. If the switch is stacked stop here and let me know.

- Get a maintenance window (as we have to reboot the switch)
- Get a backup of your current running-config / boot.cfg (download it to your computer via FTP)
- Get a copy of the boot.params file located on the /flash (download it to your computer via FTP)
- Get the up2date AOS 6.3.1.1138.R01 from the web for later install (makes no sense to install the old outdated AOS code again)
- Do NOT power off the switch during the process
- Read through the whole procedure and get familiar with it before starting

I assume you downloaded the boot.cfg / boot.params and you are ready for a network maintenance window ...

Let's get started ...

Format the flash:

Switch-> newfs /flash

In case you get an error message that "you have to mount /uflash first" we need to use the dshell function call (this bug is also resolved in later MRs):

Just do this if the newfs /flash failed!
Switch-> dshell
Working: [Kernel]-> diskInit "/flash"
Working: [Kernel]-> exit
Switch->

The flash is totally empty now. We need to create two folders:

Switch-> mkdir "working"
Switch-> mkdir "certified"

Now it is time to upload the new AOS code via FTP (6.3.1.1138.R01)

(I won't describe this in detail as I assume you know how to do this. Upload the images to the "/flash/working" directory of the switch in BINARY mode!)

Now you upload the boot.cfg to the "/flash/working" directory in ASCII mode and the boot.params to the "/flash" in ASCII mode.

Issue the following command to load from the new AOS:

Switch-> reload working no rollback-timeout

It is very important to use the command and not just power-cycle!

Your OmniSwitch will reload from the working directory now and start up the new AOS. The boot will take some more time as usual as the SSH key needs to be generated (which takes some time).

As the userTable was deleted during the format you need to connect with the admin/switch default user and add all your user accounts. Don't forget to set a secure admin password.

After the switch booted up and you got the "NIs are ready"-message (meaning you reach the switch again via SSH/Telnet) you just need to issue the following command to synchronize the folders:

Switch-> copy working certified

Please let me know in case you have further questions.

-benny

[orochi_y]

Hi Benny,

Thank for the comprehensive clarification and solution. I'm currently looking for the available AOS for 6850 which can be download because the "Purchased Software Download Server" only limited me to download version 6.3.1.871 R01.

In fact, i'm loaning another backup units while i'm implementing the formatting.

Will let you know what is the out come.

cheers.
Orochi.

[benny]

Hi Orochi,

The Business Partner who sold you the OmniSwitches has access to the Alcatel-Lucent BPWS (Business Partner WebSite). The BP can download the AOS there. Just ask your BP to provide you with the latest OS6850 6.3.1.R01 Maintenance Release (Build 1138 is the most recent one).

Good luck!

-benny

[orochi_y]

Hi Benny,

Great to see you online at this time. I'm currently doing the "reformatting" of the /flash, but unfortunately the system reject the command that i've input:

-> newfs /flash
/uflash must be mounted to perform a newfs.
ERROR: Call to newfs failed.
-> dshell
Certified: [Kernel]->diskinit "/flash"
undefined symbol: diskinit
Certified: [Kernel]->
Certified: [Kernel]->diskinit /flash
undefined symbol: diskinit
Certified: [Kernel]->exit

And the command available from the AOS as below:

help Print this list
ioHelp Print I/O utilities help info
dbgHelp Print debugger help info
nfsHelp Print nfs help info
netHelp Print network help info
spyHelp Print task histogrammer help info
timexHelp Print execution timer help info
h [n] Print (or set) shell history
i [task] Summary of tasks' TCBs
ti task Complete info on TCB for task
sp adr,args... Spawn a task, pri=100, opt=0x19, stk=20000
taskSpawn name,pri,opt,stk,adr,args... Spawn a task
td task Delete a task
ts task Suspend a task
tr task Resume a task
d [adr[,nunits[,width]]] Display memory
m adr[,width] Modify memory
mRegs [reg[,task]] Modify a task's registers interactively
pc [task] Return task's program counter

Type <CR> to continue, Q<CR> to stop: d

iam "user"[,"passwd"] Set user name and passwd
whoami Print user name
devs List devices
ld [syms[,noAbort][,"name"]] Load stdin, or file, into memory
(syms = add symbols to table:
-1 = none, 0 = globals, 1 = all)
lkup ["substr"] List symbols in system symbol table
lkAddr address List symbol table entries near address
checkStack [task] List task stack sizes and usage
printErrno value Print the name of a status value
period secs,adr,args... Spawn task to call function periodically
repeat n,adr,args... Spawn task to call function n times (0=forever)
version Print VxWorks version info, and boot line

i notice the approach you sent to me is under "working:[kernel]", but mine is under "certified:[kernel]"

could this be the problem?

regards,
orochi

[benny]

Hi Orochi,

The dshell uses pure function calls and therefore is case-sensitive.

Sorry I forgot to mention that.

Switch-> dshell
Working: [Kernel]-> diskInit "/flash"
Working: [Kernel]-> exit

It is a capital "i" ...

-benny

[orochi_y]

Hi Benny,

Thanks for that to pin-point out my mistaken also. I've just successfully "refresh" the /flash:

-> fsck /flash
/flash/ - disk check in progress ...
/flash/ - Volume is OK
total # of clusters: 62,316
# of free clusters: 17,145
# of bad clusters: 0
total free space: 17,556,480
max contiguous free space: 17,541,120 bytes
# of files: 27
# of folders: 5
total bytes in files: 46,230,574
# of lost chains: 0
total bytes in lost chains: 0


The lost chain has finally disappear. I'm now monitoring the status for that whether will it still hang un-attended.

You're great bro.
Thanks again.
Orochi