0

I am currently finding myself in a position where I need to design a local network for parallel processing of large amounts of data from multiple sources. Unfortunately I lack the experience to predict possible bottlenecks, so I hope that you can help me. If this is the wrong forum, please tell me where to post this instead.

So the problem is as follows: I have anywhere from 5 to 15 USB-sticks with up to 1TB each (but most commonly around ~500GB) of video data. I need to process these video files, and combine the results to one location (external HDD).

The goal is to allow one hub/computer to divide the video files across multiple (N) servers to allow them to process these files and send them back.

My current idea is as follows:

[USB-hub] -> [Main computer] ------> [network switch] ----[Nx1gbit cable]-----> [N servers]

The Main computer has a network card to allow for 10gbit output. Connected to this computer is a USB3 hub to allow multiple USB drives to send data. The computer is connected to a network switch with a CAT6a cable to allow for 10gbit transfer speeds.

The network switch allows the computer to transfer the files to the servers to allow for concurrent processing. This switch should have at least 1x10gbit and Nx1gbit connection Due to the storage of these servers (HDD) being a bottleneck, I estimate that 1gbit should be fast enough.

I fear for the following bottlnecks, and I have some ideas to prevent them, but I would like your help:

  • The USB hub as a bottleneck, since all connected USB drives connect through one USB3 port.
  • Overhead on transferring files from usb drive to network. I don't know if the overhead to transfer files directly from USB storage over network to HDD drives has any overhead for the CPU or if it requires a certain amount of RAM to buffer files.

Are these predicted bottlenecks correct? If so, do you have any advice on how to solve them?

Thanks in advance

1 Answer 1

2

USB 3 HDDs can exceed 200 MB/s which is nearly twice 1G Ethernet speed. Normal flash sticks are far slower though.

USB 3.2 tops out at more than 1.5 GB/s, even exceeding 10G Ethernet bandwidth. Further details are off-topic here, I'd recommend testing on your platform.

Overhead for network transfer depends on the protocol you're using. Most commonly, protocols are based on TCP over IPv4 which should top out at slightly more than 115 MB/s for 1G Ethernet (1000BASE-T).

Local overhead (CPU/RAM) entirely depends on the platforms, tools and protocols at hand and are altogether off-topic here, sorry. Capacity planning could be on topic on Server Fault but they'll want to see much more detail (hardware, software, settings) for an on-topic business network.

1
  • Thanks for your fast and concise response, I feel more reassured now. The flash sticks currently in use output roughly 85-90MB/s and I don't see an alternative coming up any time soon, so I think I will go ahead with this design. Edit: Also thanks for correcting me on HDD being the bottleneck, which isn't the case. Jan 11 at 12:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.