I am currently finding myself in a position where I need to design a local network for parallel processing of large amounts of data from multiple sources. Unfortunately I lack the experience to predict possible bottlenecks, so I hope that you can help me. If this is the wrong forum, please tell me where to post this instead.
So the problem is as follows: I have anywhere from 5 to 15 USB-sticks with up to 1TB each (but most commonly around ~500GB) of video data. I need to process these video files, and combine the results to one location (external HDD).
The goal is to allow one hub/computer to divide the video files across multiple (N) servers to allow them to process these files and send them back.
My current idea is as follows:
[USB-hub] -> [Main computer] ------> [network switch] ----[Nx1gbit cable]-----> [N servers]
The Main computer has a network card to allow for 10gbit output. Connected to this computer is a USB3 hub to allow multiple USB drives to send data. The computer is connected to a network switch with a CAT6a cable to allow for 10gbit transfer speeds.
The network switch allows the computer to transfer the files to the servers to allow for concurrent processing. This switch should have at least 1x10gbit and Nx1gbit connection Due to the storage of these servers (HDD) being a bottleneck, I estimate that 1gbit should be fast enough.
I fear for the following bottlnecks, and I have some ideas to prevent them, but I would like your help:
- The USB hub as a bottleneck, since all connected USB drives connect through one USB3 port.
- Overhead on transferring files from usb drive to network. I don't know if the overhead to transfer files directly from USB storage over network to HDD drives has any overhead for the CPU or if it requires a certain amount of RAM to buffer files.
Are these predicted bottlenecks correct? If so, do you have any advice on how to solve them?
Thanks in advance