Hello, I've noticed our rsyncs of fedora sometimes end up crawling along at around 50-150kB/s. When there's a lot of content to mirror, this means the sync could take several days, or even a week - although I'd imagine the remote end would terminate the connection long before that. In normal circumstances, we'd get anything between 5-10MB/s from dl.fedoraproject.org. I was thinking that a particular host in pool of servers was having an issue, so I did an rsync from it directly, however the speed was normal:

rsync -Pav \ 209.132.181.26::fedora-buffet0/fedora/linux/releases/24/Workstation/x86_64/iso/Fedora-Workstation-netinst-x86_64-24-1.2.iso . ---8<--- receiving incremental file list Fedora-Workstation-netinst-x86_64-24-1.2.iso 460324864 100% 6.61MB/s 0:01:06 (xfer#1, to-check=0/1) sent 55 bytes received 460381543 bytes 6720899.24 bytes/sec total size is 460324864 speedup is 1.00 These slow downs only seem to occur when we use dl.fedoraproject.org, if we switch to another host such as download-i2.fedoraproject.org, for example, it remains stable. dl.fedoraproject.org is around 150ms away from us (connecting from Dublin, Ireland), but I wouldn't imagine this would cause such drastic speed drops.

Hi Stephen, On Thu, 25 Aug 2016 13:17:33 -0400 Stephen John Smoogen <smooge(a)gmail.com&gt; wrote: > OK the load average on the servers is a bit up but the NFS throughput > looks about normal.. so I am not sure what is causing it yet. I will > investigate more on my side. > > Could you look on your side for the following: > 1) Can you give us an mtr from your host to dl05.fedoraproject.org (it > is one of the dl servers). Run for about 10 minutes. > 2) Do this again while doing an rsync. Send us both at the end of > those runs. 3) Can you try doing an rsync from the various dl servers > to see if there is one which is slow > Thanks for looking into it, I'm attaching an mtr to dl05 with a 10 minute run, along with one when rsync is running.

A log of rsyncs from the 5 servers in the dl.fedoraproject.org rotation is also attached.

> > ms round trip time has little bearing anymore in bandwidth issues :(. > You can have an mid-stream which is throttling you down for some > reason (they don't think the next guy is paying them enough) or you > can have a bad cable somewhere which pings perfectly nicely but stops > up. Yeah I didn't think so, we provide internet access for the majority of academic institutions in Ireland, so our connectivity is usually pretty good :) rg -- Rob Gallagher | Public Key: 0x841386E3 HEAnet Limited, Ireland's Education and Research Network 1st Floor, 5 George's Dock, IFSC, Dublin D01 X8N7 Registered in Ireland, no 275301 ☎ (+353-1) 6609040 WWW: http://www.heanet.ie/

Hi Stephen, There's actually an rsync running at the moment which is exhibiting the slowness issue, the sync is happening from from dl03 (209.132.181.25), and it's showing speeds around 100kB/s. We're connecting from 193.1.219.88. rg On Fri, 26 Aug 2016 10:43:14 -0400 Stephen John Smoogen <smooge(a)gmail.com&gt; wrote: > On 26 August 2016 at 07:21, Rob Gallagher > <robert.gallagher(a)heanet.ie&gt; wrote: > > Hi Stephen, > > > > On Thu, 25 Aug 2016 13:17:33 -0400 > > Stephen John Smoogen <smooge(a)gmail.com&gt; wrote: > > > >> OK the load average on the servers is a bit up but the NFS > >> throughput looks about normal.. so I am not sure what is causing > >> it yet. I will investigate more on my side. > >> > >> Could you look on your side for the following: > >> 1) Can you give us an mtr from your host to dl05.fedoraproject.org > >> (it is one of the dl servers). Run for about 10 minutes. > >> 2) Do this again while doing an rsync. Send us both at the end of > >> those runs. 3) Can you try doing an rsync from the various dl > >> servers to see if there is one which is slow > >> > > > > Thanks for looking into it, I'm attaching an mtr to dl05 with a 10 > > minute run, along with one when rsync is running. > > > > I like how to get to the UK from ireland it takes 180 ms but it takes > 150ms to get to the US after going through there.. ICMP flow control > is wonderful :). (Or Brexit affects packets.. take your pick). > > > A log of rsyncs from the 5 servers in the dl.fedoraproject.org > > rotation is also attached. > > > > OK dl01 and dl02 seem to be the problem ones in your check. I will > look them over and see what is up. Could you try dl04 for a bit and > see if you have a better set of runs? > > >> > >> ms round trip time has little bearing anymore in bandwidth > >> issues :(. You can have an mid-stream which is throttling you down > >> for some reason (they don't think the next guy is paying them > >> enough) or you can have a bad cable somewhere which pings > >> perfectly nicely but stops up. > > > > Yeah I didn't think so, we provide internet access for the majority > > of academic institutions in Ireland, so our connectivity is usually > > pretty good :) > > > > rg > > > > -- > > Rob Gallagher | Public Key: 0x841386E3 > > > > HEAnet Limited, Ireland's Education and Research Network > > 1st Floor, 5 George's Dock, IFSC, Dublin D01 X8N7 > > Registered in Ireland, no 275301 > > ☎ (+353-1) 6609040 WWW: http://www.heanet.ie/ > > > > > -- Rob Gallagher | Public Key: 0x841386E3 HEAnet Limited, Ireland's Education and Research Network 1st Floor, 5 George's Dock, IFSC, Dublin D01 X8N7 Registered in Ireland, no 275301 ☎ (+353-1) 6609040 WWW: http://www.heanet.ie/