{{ message }}
Repository navigation
Wondering about mbuffer with zfs send / recv #86
Description
Hello,
Found the project by looking at the next dev summit from OpenZFS.
I quickly looked at the code, and wondered whether zrepl can use a buffer when sending / receiving data.
Having compared basic zfs send / recv and buffered one via mbuffer, I was able to get 4Gbps instead of 2.5Gbps between two TrueNAS units, ie:
Receiver:
mbuffer -s 128k -m 1G -4 -I 5005 | zfs receive pool/dataset
Sender:
zfs send -R pool@dataset@snapshotsend | mbuffer -s 128k -m 1G -O 10.13.37.0:5005
Does zrepl use a buffer mechanism, or is there space for improving when mbuffer is detected ?
Best regards.
Activity
dch commentedon Aug 9, 2018
deajan commentedon Aug 9, 2018
@dch Didn't add ssh to my tests since I dealt with lan copies, but I guess it would be the real bottleneck as always.
Had other ssh speed issues, when using sshfs, where I was able to go from 250Mbps to 600Mbps using a "lighter" ssh cipher, aes128-ctr, which is also AES-NI supported in order to offload the cpu, even in most hypervisors.
So I do wonder if hpenc would be faster than a tuned ssh.
A good performance comparaison could be:
Nevertheless, since zfs send streams can have changing bandwith usage, and sender or receiver could be temporarily blocked by IO, using mbuffer should in most cases significantly improve overall speed.
In my case, most of the time, 1GB buffers were loaded round 0-30% with 99% spikes, on a 10Gbps network between two TrueNAS units with RAID Z2 10 disk arrays, which only activity at the time was zfs replication.
Btw, I'm sorry but at the time my TrueNAS units are geo-replicated with a 1Gbps link only, so it wouldn't make sense to make a benchmark since now my bottleneck is network.
problame commentedon Oct 17, 2018
At first, sorry for the late reply.
Generally, zrepl buffers internally to chunk the zfs send stream for the various transports.
You can experiment with buffering by placing a wrapper shell script into the PATH used by the zrepl daemon, pass through all commands except
zfs sendto the real zfs(8) binary and use mbuffer or similar in case the wrapper script is invoked with thesendsubcommand.deajan commentedon Oct 19, 2018
Nice idea... Will do so when I'll have spare time, or next time I need quick zfs send / receive :)
asche77 commentedon Mar 26, 2020
Would be great if mbuffer could be included - especiallly small send/receive (e.g. zero size snapshots) seem to be sent sequentially and this really hits the transfer rate (dropping down to a few kB).
I have used mbuffer with a few other zfs sends and really does speed up transfers.
problame commentedon Mar 26, 2020
@asche77I think that the latency for small incremental transfers won't improve much with mbuffer for zrepl. However, work is underway to parallelize replication - I already use it on my setup, but didn't get around going through all the corner cases just yet. probably subscribe to #140
Add info about some of resolved upstream issues