We are currently designing a system to securely upload very large files (>10GB) across the internet to a storage server at our office. The bottleneck in this system will no doubt be the remote uplink speed. Clients will join a VPN to access the server. What is the most efficient way to make these uploads? We're looking at Nextcloud, NFS, FTP, rsync, etc. but I'm not sure what the overheads involved here are or how to calculate them.

  • Welcome to Network Engineering! Your Internet bandwidth will be the limiting factor by far, greatly overshadowing any minor differences in the applications. Even assuming 100Mb connections by your clients, it will take over 22 hours to upload a 10Gb file
    – Ron Trunk
    Nov 15 '18 at 13:35
  • @RonTrunk You might want to recalculate that... 10 GB over 100 Mbit/s take only ~15 min. For 1 Mbit/s you'd have ~24 h.
    – Zac67
    Nov 15 '18 at 13:48
  • Hello Loop and welcome to NE. I'd just suggest you revisit your assumption that the limiting factor will be uplink speed; it may well be RTT. I suggest avoiding anything block-based (CIFS, NFS, TFTP) for this reason. I had this exact issue for a client recently whose data had to travel a very laggy ~300ms RTT line for a 300Gbyte tree (many 1~2Gbyte files, median 100Kbyte), and we had most success with a) rsync over ssh for server to server, b) nextcloud over https for server-to-laptop. We skipped the VPN as it introduced even more lag.
    – jonathanjo
    Nov 15 '18 at 13:59
  • 1
    "not sure what the overheads involved here are or how to calculate them." respecfully I'd say do experiments as close as you can to your live case, rather than any amount of calculations other than theoretical minimums.
    – jonathanjo
    Nov 15 '18 at 14:01
  • @Zac67 You're right. That's what I get for trying to answer before drinking coffee!
    – Ron Trunk
    Nov 15 '18 at 14:18

Since VPN tunnels encapsulate the original network packets, they create overhead - the size of the encapsulating headers. Accordingly, a VPN using small headers provides better efficiency, higher net speed or lower effective data volume per user net data volume.

Basically, if the application is able to provide decent security (esp. by using TLS) and authentication you could omit the VPN tunnel and its overhead.

Which application protocol is the most efficient for your use case depends on the exact workload. FTP/FTPES is efficient for large, unique files but the protocol overhead may be a no-go for numerous, small transfers. NFS may be negatively affected by high network latency. rsync is highly efficient for transmitting file changes between locations, and so on.

Not the answer you're looking for? Browse other questions tagged or ask your own question.