LACP configuration with Broadcom BACS: oper state down

[chaos_prevails]

Hello,

we have two stacked together omniswitches 6400-P24 and I would like to configure dynamic Link Aggregation 802.3ad for some servers to improve speed between those servers. They all have two 1Gb/s broadcom NICs running the Broadcom Advanced Control Suite 3 (running Windows 2008 R2).

On the switch I have the following configuration:
lacp linkagg 1 size 2 admin state enable
lacp linkagg 1 name "aggr-no1"
lacp agg 1/1 standby enable
lacp agg 1/1 actor admin key 1
lacp agg 2/1 standby enable
lacp agg 2/1 actor admin key 1

on the server I configured two NICS as one team without VLAN. In the Information tab I get the following info:
==Team ==
Team Type: 802.3ad
Driver Status: loaded
Link Status: up
Offload Capabilities: LSO, CO
Mtu: 0

However the first NIC (connected to port 1/1) has the BASP State: Not Active.

On the switch side I see the following information:

Port 1/1:
Admin State: enabled
Opper State: Down !<-- shouldn't this be "Up"
Port State: Selected !<-- shouldn't this be "Aggregated"
Link State: Up
Primary: no
Under the Partner Oper System ID I see the correct MAC Address (of the aggregated Broadcom NIC)

Port 1/2:
Admin State: enabled
Opper State: Up
Port State: Attached !<-- shouldn't this be "Aggregated"
Link State: Up
Primary: Yes

I get transfer speeds of around 60MB/s (simple copy-paste). When I measure the speed with Smart Load balancing and Failover (configured in BACS, no configuration on the Alcatel Switch), I get around 110MB/s. With LACP I would expect at least the same (for more the end point would have LACP correctly configured as well I guess).

I tried finding information how to correctly set up LACP and tried using a additional VLAN for the two aggregated ports, having or not having the same VLAN ID in the BACS configuration. But as soon as I set a specific VLAN ID on the BACS I cannot access the server any more.

What strikes me is also the Port State of both ports: there is a port stated "aggregated". Shouldn't this be the port state of the two ports? Instead I see "Selected" and "Attached"

[devnull]

I don't know Broadcom BACS, but yes on switch side both ports must be attached if LACP is working fine.

# show linkagg 1 port

Slot/Port Aggregate SNMP Id Status Agg Oper Link Prim
---------+---------+-------+----------+----+----+----+----
1/49 Dynamic 1049 ATTACHED 1 UP UP YES
2/49 Dynamic 2049 ATTACHED 1 UP UP NO

I must say i have never seen the use of "standby enable" are you sure you need that?

i have configured

lacp linkagg 1 size 2 admin state enable
lacp linkagg 1 name "2CoreSwitch"
lacp linkagg 1 actor admin key 1
lacp agg 1/49 actor admin key 1
lacp agg 2/49 actor admin key 1

So you are missing the actor key for the linkagg
"lacp linkagg 1 actor admin key 1"

What VLANs are configured on that linkagg?
"show vlan port 1" (x being linkagg number)

the setup must match the one on the other side if your are using "only the default vlan" you may need to change that on switch side

vlan XY port default 1

or if you want to add a VLAN as tagged additional
vlan XY 802.1q 1

This must be reflected on Server side.

[chaos_prevails]

hello devnull,

thank you, the admin key for the link aggregation was indeed missing.

the configuration is now (I now used linkagg 3 instead of 1):
lacp linkagg 3 size 2 admin state enable
lacp linkagg 3 name "XXXX"
lacp linkagg 3 actor admin key 3
lacp agg 1/3 standby enable (<< automatically added when I add this port via the webinterface)
lacp agg 1/3 actor admin key 3
lacp agg 2/3 standby enable (<< automatically added when I add this port via the webinterface)
lacp agg 2/3 actor admin key 3

on the server side I recreated the 802.3ad link aggregation but it still doesn't work (one link is still having port state "Selected" and on the server side I still see "BASP State Not Active".

So I added a VLAN for this link aggregation. First on the server I added a tagged vlan to the link aggregation with id 13. (untagged is also possible but it also didn't work).

Then on the switch:
vlan 13 enable name "lagg_XXX"
vlan 13 port default 3

Didn't work, but this VLAN is without tags. So I thought I have to add tagging:

vlan 13 802.1q 3
I get:
ERROR: Port cannot be tagged in default VLAN.

so apparently the ports (or the aggregation) needs to be removed from de default VLAN. If I do
show vlan port 1/3
show vlan port 2/3
I get no result.

But if I do
show vlan port 3
I get:
vlan type status
--------+---------+--------------
13 default forwarding

so the type = default is the culprit?

But how do I remove the default VLAN 1 from the link aggregation?

[devnull]

With
"vlan 13 port default 3" you put vlan 13 as default/untagged/native vlan on linkagg3 (the physical ports 1/3 and 2/3 are no longer used as this is linkagg 3 now).

As you have seen:

Code: Select all

show vlan port 3

 vlan type status
--------+---------+--------------
 13 default forwarding

the native/default vlan is vlan 13.

you can't have the same vlan tagged and untagged on a port/linkagg, so if you need to have vlan 13 tagged (instead of default) you need to first use another native vlan.

e.g.
vlan 1 port default 3
vlan 13 802.1q 3

now you should get something like

Code: Select all

show vlan port 3

 vlan type status
--------+---------+--------------
 1 default forwarding
 13 qtagged forwarding

Now all traffic for vlan 13 is getting a tag when "leaving" the linkagg, while all untagged traffic is expected to be in vlan 1.

Still this is not the solution with your lacp config.
standby enable is imho a keyword for DHL (Dual Homeing link) - if you have a lacp on the other side i would try to get rid of this lines.

[chaos_prevails]

Thank you devnull,

removing the standby enable option from the ports solved the problem. the links went immediately online without any delay (which was the case when I had the standby enable option). for some reason this option is automatically added when creating the link aggregation via de webinterface. after I did everything via de cli it worked out of the box.

the configuration on the switch is as follows (no vlan or any other configuration necessary):

Code: Select all

...
! Link Aggregate :
lacp linkagg 1 size 2 admin state enable
lacp linkagg 1 name "x1"
lacp linkagg 1 actor admin key 1
lacp linkagg 2 size 2 admin state enable
lacp linkagg 2 name "x2"
lacp linkagg 2 actor admin key 2
lacp linkagg 3 size 2 admin state enable
lacp linkagg 3 name "x3"
lacp linkagg 3 actor admin key 3
lacp linkagg 4 size 2 admin state enable
lacp linkagg 4 name "x4"
lacp linkagg 4 actor admin key 4
lacp agg 1/1 actor admin key 1
lacp agg 1/2 actor admin key 2
lacp agg 1/3 actor admin key 3
lacp agg 1/4 actor admin key 4
lacp agg 2/1 actor admin key 1
lacp agg 2/2 actor admin key 2
lacp agg 2/3 actor admin key 3
lacp agg 2/4 actor admin key 4
! VLAN AGG:
...

show linkagg 1 port
gives:

Code: Select all

Slot/Port Aggregate SNMP Id   Status   Agg  Oper Link Prim Standby
---------+---------+-------+----------+----+----+----+----+-------
   1/1    Dynamic     1001  ATTACHED     1  UP   UP   YES  NO
   2/1    Dynamic     2001  ATTACHED     1  UP   UP   NO   NO

on the server side I configured Link aggregation 802.ad as team type without any vlan or further configuration.

regarding the speed enhancements: when copying a big file the speed increased on average by probably +10%, not more. in the beginning of the file transfer I get peaks of 190MB/s - probably due to caching of the file? I thought already that it is hard to test and that drive speeds and load on the servers will play a big part.

Well, at least I got it configured and hopefull this post can also help others. If there are any ways to improve speed by a configuration setting on the switch, I would like to hear about.

[devnull]

I would do a speed test with jperf or a load generator.

Always remember: a single session is not load balanced on lacp, lacp does a per-source MAC(per sorce/destination pair?) hashing, so a single connection is expected to NOT execess 120MB/second (Gigabit) while having several parallel sessions of other host-pairs will load balance fine.

Hence: One Backup-Server -> one Storage will be using one link only.
Multiple PCs -> One Backup-Server will in total be up to n (number of links in linkagg) * 1Gbit (linkspeed)
You can check
"hash-control extended" to enable TCP/UDP port based hashing -> Still one session will be on one link maximum.

This is a expected behavior on all Switches (all verndors) using lcap. (Ok there maybe a better solution with Brocade VCS http://blog.ipspace.net/2011/04/brocade ... rfect.html)