Ping. == Steve Tyler stephent98(a)hotmail.com Portland, OR, USA. From: Matt_Domsch(a)Dell.com To: bill.nickless(a)pnl.gov; timw(a)pnl.gov CC: stephent98(a)hotmail.com; mirror-admin(a)fedoraproject.org Date: Thu, 7 Apr 2011 11:54:45 -0500 Subject: FW: mirror.pnl.gov stale Bill, Tim, FYI… --Matt DomschTechnology StrategistDell | Office of the CTO From: mirror-admin-bounces(a)fedoraproject.org [mailto:mirror-admin-bounces@fedoraproject.org] On Behalf Of Steve Tyler Sent: Thursday, April 07, 2011 6:51 AM To: mirror-admin(a)fedoraproject.org Subject: mirror.pnl.gov stale mirror.pnl.gov seems to be stale, again. http://mirror.pnl.gov/fedora/linux/development/15/x86_64/os/repodata/ repomd.xml 2011-Apr-03 16:06:38 http://mirror.pnl.gov/fedora/linux/updates/testing/15/x86_64/repodata/ repomd.xml 2011-Apr-02 20:55:36 == Steve Tyler stephent98(a)hotmail.com Portland, OR, USA.

Thanks for looking into this, Bill. Still no joy, though ... http://mirror.pnl.gov/fedora/linux/development/15/x86_64/os/repodata/ http://mirror.pnl.gov/fedora/linux/updates/testing/15/x86_64/repodata/ Compare: http://fedora.osuosl.org/linux/development/15/x86_64/os/repodata/ http://fedora.osuosl.org/linux/updates/testing/15/x86_64/repodata/ == Steve Tyler stephent98(a)hotmail.com Portland, OR, USA. From: bill.nickless(a)pnl.gov To: stephent98(a)hotmail.com; matt_domsch(a)dell.com; timw(a)pnl.gov CC: mirror-admin(a)fedoraproject.org Date: Thu, 21 Apr 2011 11:42:49 -0700 Subject: RE: mirror.pnl.gov stale Thanks for keeping after us. It turns out that we added IPv6 support to the subnet serving MIRROR.PNL.GOV, so MIRROR started trying to rsync with its upstream using IPv6 instead of IPv4. I just sent a note to our upstream asking them to accept our IPv6 address, and the system manager there fixed it within 2 minutes. Rsync is running happily now. From: Steve Tyler [mailto:stephent98@hotmail.com] Sent: Wednesday, April 20, 2011 2:26 AM To: matt_domsch(a)dell.com; Nickless, Bill; Witteveen, Tim Cc: mirror-admin(a)fedoraproject.org Subject: RE: mirror.pnl.gov stale Ping. == Steve Tyler stephent98(a)hotmail.com Portland, OR, USA. From: Matt_Domsch(a)Dell.com To: bill.nickless(a)pnl.gov; timw(a)pnl.gov CC: stephent98(a)hotmail.com; mirror-admin(a)fedoraproject.org Date: Thu, 7 Apr 2011 11:54:45 -0500 Subject: FW: mirror.pnl.gov staleBill, Tim, FYI… --Matt DomschTechnology StrategistDell | Office of the CTO From: mirror-admin-bounces(a)fedoraproject.org [mailto:mirror-admin-bounces@fedoraproject.org] On Behalf Of Steve Tyler Sent: Thursday, April 07, 2011 6:51 AM To: mirror-admin(a)fedoraproject.org Subject: mirror.pnl.gov stale mirror.pnl.gov seems to be stale, again. http://mirror.pnl.gov/fedora/linux/development/15/x86_64/os/repodata/ repomd.xml 2011-Apr-03 16:06:38 http://mirror.pnl.gov/fedora/linux/updates/testing/15/x86_64/repodata/ repomd.xml 2011-Apr-02 20:55:36 == Steve Tyler stephent98(a)hotmail.com Portland, OR, USA.

They may simply be getting caught in the tiered propagation delay. New content hit the masters here: Starting umdl 2011-04-22T00:00:02.595742 pub/fedora/linux/development/15/x86_64/os/Packages has changed mirror.pnl.gov checked in with report_mirror (so MM thinks they're completely up-to-date) < 3 hours later: Last Checked In: 2011-04-22 02:40:51 Yet less than an hour later, MM thinks they're stale again, which means they didn't pick up the new content yet, though they thought they were up-to-date and ran report_mirror to tell MM they were up to date. But the crawler sees the mistake and marks them stale anyhow. This can happen if it takes >4 hours for the content to propagate... Starting crawl 2011-04-22T03:26:30.765103 Directory pub/fedora/linux/development/15/x86_64/os/Packages is not up-to-date on this host. And they stay marked stale for 10 hours, until the crawler sees they are up-to-date again: Starting crawl 2011-04-22T13:27:44.664182 http://mirror.pnl.gov/fedora/linux/development/15/x86_64/os/Packages and they stay up-to-date for 10 hours... Starting crawl 2011-04-22T23:26:18.829286 http://mirror.pnl.gov/fedora/linux/development/15/x86_64/os/Packages Then the content on the master mirror was updated: Starting umdl 2011-04-22T23:30:03.406581 pub/fedora/linux/development/15/x86_64/os/Packages has changed but it had not yet propagated to the pnl.gov mirror 2 hours later, so they get marked stale again... Directory pub/fedora/linux/development/15/x86_64/os/Packages is not up-to-date on this host. Ending crawl 2011-04-23T01:42:48.665619 Bill, who are you pulling from, and how frequently? (yes, I really wish we had something better than cronjob'd rsyncs, exactly for this reason. I'm hopeful the CHASM project will give us that replacement, but it's been slow going...) So Steve, I'm not sure I'd paint it as insufficient admin attention, more likely simply need to increase the rsync frequency for such a high-bandwidth-advertised mirror, and for whomever you're pulling from also. -- Matt Domsch Technology Strategist Social Media and Community Professional Dell | Office of the CTO From: Steve Tyler [mailto:stephent98@hotmail.com] Sent: Friday, April 22, 2011 9:03 PM To: bill.nickless(a)pnl.gov; Domsch, Matt; timw(a)pnl.gov Cc: mirror-admin(a)fedoraproject.org Subject: RE: mirror.pnl.gov stale Hi Bill and Matt, "Thanks for keeping after us." Your site is still stale. :-( Matt: Isn't there a way to take pnl.gov offline? (They clearly cannot maintain a mirror ...) == Steve Tyler stephent98@hotmail.com<mailto:stephent98@hotmail.com> Portland, OR, USA. ________________________________ From: stephent98(a)hotmail.com To: bill.nickless(a)pnl.gov; matt_domsch(a)dell.com; timw(a)pnl.gov CC: mirror-admin(a)fedoraproject.org Subject: RE: mirror.pnl.gov stale Date: Fri, 22 Apr 2011 02:22:02 -0700 Thanks for looking into this, Bill. Still no joy, though ... http://mirror.pnl.gov/fedora/linux/development/15/x86_64/os/repodata/ http://mirror.pnl.gov/fedora/linux/updates/testing/15/x86_64/repodata/ Compare: http://fedora.osuosl.org/linux/development/15/x86_64/os/repodata/ http://fedora.osuosl.org/linux/updates/testing/15/x86_64/repodata/ == Steve Tyler stephent98@hotmail.com<mailto:stephent98@hotmail.com> Portland, OR, USA. ________________________________ From: bill.nickless(a)pnl.gov To: stephent98(a)hotmail.com; matt_domsch(a)dell.com; timw(a)pnl.gov CC: mirror-admin(a)fedoraproject.org Date: Thu, 21 Apr 2011 11:42:49 -0700 Subject: RE: mirror.pnl.gov stale Thanks for keeping after us. It turns out that we added IPv6 support to the subnet serving MIRROR.PNL.GOV, so MIRROR started trying to rsync with its upstream using IPv6 instead of IPv4. I just sent a note to our upstream asking them to accept our IPv6 address, and the system manager there fixed it within 2 minutes. Rsync is running happily now. From: Steve Tyler [mailto:stephent98@hotmail.com] Sent: Wednesday, April 20, 2011 2:26 AM To: matt_domsch(a)dell.com; Nickless, Bill; Witteveen, Tim Cc: mirror-admin(a)fedoraproject.org Subject: RE: mirror.pnl.gov stale Ping. == Steve Tyler stephent98@hotmail.com<mailto:stephent98@hotmail.com> Portland, OR, USA. ________________________________ From: Matt_Domsch(a)Dell.com To: bill.nickless(a)pnl.gov; timw(a)pnl.gov CC: stephent98(a)hotmail.com; mirror-admin(a)fedoraproject.org Date: Thu, 7 Apr 2011 11:54:45 -0500 Subject: FW: mirror.pnl.gov stale Bill, Tim, FYI... -- Matt Domsch Technology Strategist Dell | Office of the CTO From: mirror-admin-bounces(a)fedoraproject.org [mailto:mirror-admin-bounces@fedoraproject.org] On Behalf Of Steve Tyler Sent: Thursday, April 07, 2011 6:51 AM To: mirror-admin(a)fedoraproject.org Subject: mirror.pnl.gov stale mirror.pnl.gov seems to be stale, again. http://mirror.pnl.gov/fedora/linux/development/15/x86_64/os/repodata/ repomd.xml 2011-Apr-03 16:06:38 http://mirror.pnl.gov/fedora/linux/updates/testing/15/x86_64/repodata/ repomd.xml 2011-Apr-02 20:55:36 == Steve Tyler stephent98@hotmail.com<mailto:stephent98@hotmail.com> Portland, OR, USA.