1

We're purchasing new servers that have Mellanox ConnectX 6 network cards in them, which accept QSFP56 transceivers. The card says it supports 10/25/40/50/100/200G. However, our switching backbone is all 10G with SFP+ transceivers.

If I put a 40G transceiver in the card, can I directly connect that, 1:1, to the 10G transceivers in my Cisco 4500X switches, and the 40G transceiver will be able to run at 10G? Or, do I need to get a QSFP to SFP adapter and put SFPs in the Mellanox cards? I'd rather just buy 40G transceivers, now, if possible, and be able to use them when we eventually upgrade the switching backbone, rather than buying 10G transceivers and adapters, and then buying 40G transceivers later.

I'm aware of the existence of breakout cables, but I don't have enough ports for that, and we really don't want to buy a new switch if we can avoid it.

New switches are in the cards, but not for a couple more quarters, at least, due to budgets.

Improve this question
1
  • You should use a 10G transceiver which is guaranteed to work. A 40G transceiver will not work unless its datasheet and the switch's datasheet explicitly state that that is the case.
    – Zac67 ♦
    Commented Feb 21, 2022 at 7:07

1 Answer 1

Reset to default
5

The only way you'd be able to keep using the same 40G QSFP is to get a 40G-SR4 and break it out to 4x10GE coming from the NIC and then connect one of those 10G's to your 4500.

Once you have a switch that can support QSFP you could then connect it with an MPO-12 fiber cable to another 40G-SR4 in the switch and run it as a single 40G link.

You'd need to check with Mellanox directly to determine if that NIC supports an SFP+ adapter, as it isn't immediately clear from what's published.

Improve this answer
5
  • From what I've seen elsewhere, I'm led to believe all 4 of the breakout 10G cables have to be connected and up or the 40G interface will not come up. Is this correct?
    – dodexahedron
    Commented Feb 21, 2022 at 8:43
  • @dodexahedron: not generally. If the NIC, and its driver/kernel module support being "broken up" into 4x10G (presenting themselves to the upper layer as 4 independent 10G NICs), then you should not need to wire all 4 pairs of a breakout cable. You'll have to figure out the mapping of breakout pairs 1-4 to the given "sub-NIC", of course. At least that's what we do regulary on 40G Network devices. As far as server hardware and software is concerned, some other corners of stackexchange.com might be more appropriate.
    – Marc 'netztier' Luethi
    Commented Feb 21, 2022 at 14:25
  • Thanks. The software side should be easy enough to figure out. Now I just need to talk to nvidia to get more details on what these cards can actually physically do, since the documentation is pretty lacking. I'm also considering switches from fs.com, but I don't know if I can trust the reviews, there, since they look pretty fake. But those prices are almost low enough that buying one and it not delivering as promised wouldn't be THAT big of a deal. 😅
    – dodexahedron
    Commented Feb 21, 2022 at 16:34
  • @dodexahedron The purchase isn't the only cost of running a switch. You invest time in getting to know the gear and when you find out that they don't work properly mid-project you're in a bind.
    – Zac67 ♦
    Commented Feb 23, 2022 at 11:08
  • @Zac67 yes, this is fair. We've decided to look into something from Arista to fit our needs. Price per performance absolutely eats Cisco's lunch, there.
    – dodexahedron
    Commented Feb 25, 2022 at 23:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.