10 Gb blade question

Question

I am debating with a colleague about a potential configuration. We are implementing a new blade solution that has a built in 10 Gb switch and two blade servers, each having 4 10 Gb NICs.

However, the rest of our environment is still 1 Gb.

The blade switch has four external 10GBASE-T ports which will auto-sense down to 1 Gb. We plan on LAGing these.

Essentially, we will have 80 Gb of NIC bandwidth on a 10 Gb switch connected with a 4 Gb inter-switch link.

I think we should manually set the internal ports to 1 Gb, but he doesn't think there will be any noticeable performance problems from the above design.

Please let me know if you have had any experience with a similar setup, or if you know of any problems we may anticipate.

Answer 1

Basically, your 40 Gb link bundles will be for fault tolerance only unless you have blade-to-blade transfers on the blade switch; even a single 10 Gb server connection is more than your total uplink bandwidth.

There are many networks and switches with mixed port speed. Cisco has a guideline based on extensive research. The access port (your server connections) to distribution port (your external connections) bandwidth ratio should be no greater than 20:1. This means that for every 20 Gb in your access port speed, you need 1 Gb uplink port speed. What you are describing (80 Gb to 4Gb) is 20:1. The recommended distribution to core ratio is much lower at 4:1.

Answer 2

You should apply QoS between blade switches and the rest of your environment, to prevent jumbo flows from starving other flows.

How it can be accomplished depends on hardware specs, but each of your 10 GbE ports facing 1 GbE switches, has 8 hardware queues (0-7, one queue per CoS value), where packets wait their turn to leave.

You have the chance to configure scheduling algorithms used by each queue (strict priority or some kind of scheduling). The queue where Ethernet frames wait to leave blade switches, can be chosen when frames enter the switch.

Server OS, hypervisors, have to mark Ethernet frames, while blade switches have to honor CoS values, trusting their counterpart.
Once marked, frames will end in the desired queue, keeping you in control.

Read about Data Center Bridging (DCB) support and configuration between blade servers and switches.

Check DCB support in 1 GbE switches: if 1 Gb switches are DCB compliant, Pause Frames directed to blade switches, will act on specific queues in case of congestion (Ethernet Priority Flow Control), while higher priority queues will continue to move Ethernet frames.

Answer 3

There is no difference. Because you have the same port on the switch with 1Gb and 10Gb bandwidth. This switch probably has a shared packet buffer which doesn't cause a microburst drops problem. But if you get a bundle switch which has separate buffers on different ports when you may get no free space in the output queue at high load moments.