Disclaimer: I'm not a network engineer; I just have to try to do a good-enough impression to solve this problem.
We have two rooms in our office. In the first, I would like to put a large copper 10-GbE switch to provide connectivity to several servers, something like this:
This switch has 48 10G-BASE-T ports and 4 40-GbE uplink ports.
In the second room, we have many devices that require 10-GbE SFP+ connectivity. It appears something like this could work:
This switch has 48 SFP+ ports and 6 40-GbE uplink ports.
Goal: To the greatest extent possible, I would like to provide 10-GbE line-rate performance between any pair of hosts in both rooms. As long as they are plugged into the same switch, this is straightforward, but I'm wondering how best to accomplish this across the two switches. I don't think I need any fancy features like QoS, VLANs, or anything like that; I just want one flat level 2 network. Can I simply connect 4 of the 40-GbE uplinks between the two switches and get ~160 Gbps of bandwidth between the two rooms?
My application consists of continuous, very high-rate UDP flows; that is, it's common for a particular flow to take between 2-8 Gbps of bandwidth on its own. That means that each host will only be communicating with 1-2 other hosts at any time, putting a limit on the total number of flows that are running simultaneously. I'm not sure whether this simplifies or complicates the issue: I need to ensure that the UDP datagrams are not reordered by any aggregation of the uplink ports.