11:13 a.m.
Hi,
Over the last couple of weeks we've put together a mirror checking tool
and a cgi that will verify the currency of the mirrorlist. Then it will
generate lists per-country based on geoip country codes as well as a
global mirror list.
The code for this tool is available here:
http://cvs.fedora.redhat.com/viewcvs/check-mirrors/?root=fedora
We've also setup a url for the cgi for use from the yum repository
definitions.
I'd like to see this added to the fedora-release package (where
the .repo files are stored) for FC6 Test 2, if possible.
[development]
name=Fedora Core - Development
mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=rawhide&a...
and for extras:
[extras-development]
name=Fedora Extras - Development Tree
mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=extras-devel&...
Thanks,
-sv
12:24 p.m.
Hi Seth, Maybe I'm missing something, or the tool isn't fully up-to-date
yet, but when I request a mirror list for rawhidex86_64
http://mirrors.fedoraproject.org/mirrorlist?repo=rawhide&arch=x86_64
I get this as the result:
# repo = rawhide country = NL arch = x86_64
http://ftp.surfnet.nl/ftp/pub/os/Linux/distr/fedora/updates/testing/5/$ARCH/
http://ftp.nluug.nl/ftp/pub/os/Linux/distr/fedora/updates/testing/5/$ARCH/
http://ftp.surfnet.nl/ftp/pub/os/Linux/distr/fedora/5/$ARCH/os/
http://ftp.nluug.nl/ftp/pub/os/Linux/distr/fedora/5/$ARCH/os/
http://ftp.surfnet.nl/ftp/pub/os/Linux/distr/fedora/5/$ARCH/os/
http://ftp.nluug.nl/ftp/pub/os/Linux/distr/fedora/5/$ARCH/os/
That looks suspiciously like the result set for fc5, and fc5 updates (2
repo's in one response?). However it doesn't look anywhere near like
../fedora/core/development/.. repo's :-)
-- Chris
-----Original Message-----
From: fedora-devel-list-bounces(a)redhat.com
[mailto:fedora-devel-list-bounces@redhat.com] On Behalf Of seth vidal
Sent: Monday, July 17, 2006 18:13
To: fedora-devel-list(a)redhat.com
Subject: mirrorlist cgi
Hi,
Over the last couple of weeks we've put together a mirror checking tool
and a cgi that will verify the currency of the mirrorlist. Then it will
generate lists per-country based on geoip country codes as well as a
global mirror list.
The code for this tool is available here:
http://cvs.fedora.redhat.com/viewcvs/check-mirrors/?root=fedora
We've also setup a url for the cgi for use from the yum repository
definitions.
I'd like to see this added to the fedora-release package (where
the .repo files are stored) for FC6 Test 2, if possible.
[development]
name=Fedora Core - Development
mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=rawhide&a...
search
and for extras:
[extras-development]
name=Fedora Extras - Development Tree
mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=extras-devel&...
h=$basearch
Thanks,
-sv
1:11 p.m.
On Mon, 2006-07-17 at 19:24 +0200, Chris Chabot wrote:
Hi Seth, Maybe I'm missing something, or the tool isn't fully
up-to-date
yet, but when I request a mirror list for rawhidex86_64
http://mirrors.fedoraproject.org/mirrorlist?repo=rawhide&arch=x86_64
I get this as the result:
# repo = rawhide country = NL arch = x86_64
http://ftp.surfnet.nl/ftp/pub/os/Linux/distr/fedora/updates/testing/5/$ARCH/
http://ftp.nluug.nl/ftp/pub/os/Linux/distr/fedora/updates/testing/5/$ARCH/
http://ftp.surfnet.nl/ftp/pub/os/Linux/distr/fedora/5/$ARCH/os/
http://ftp.nluug.nl/ftp/pub/os/Linux/distr/fedora/5/$ARCH/os/
http://ftp.surfnet.nl/ftp/pub/os/Linux/distr/fedora/5/$ARCH/os/
http://ftp.nluug.nl/ftp/pub/os/Linux/distr/fedora/5/$ARCH/os/
That looks suspiciously like the result set for fc5, and fc5 updates (2
repo's in one response?). However it doesn't look anywhere near like
../fedora/core/development/.. repo's :-)
hmm - there does appear to be a bug going on in here somewhere.
I'll take a look. thanks!
-sv
1:28 p.m.
On Mon, 2006-07-17 at 14:11 -0400, seth vidal wrote:
On Mon, 2006-07-17 at 19:24 +0200, Chris Chabot wrote:
> Hi Seth, Maybe I'm missing something, or the tool isn't fully up-to-date
> yet, but when I request a mirror list for rawhidex86_64
>
> http://mirrors.fedoraproject.org/mirrorlist?repo=rawhide&arch=x86_64
>
> I get this as the result:
>
> # repo = rawhide country = NL arch = x86_64
> http://ftp.surfnet.nl/ftp/pub/os/Linux/distr/fedora/updates/testing/5/$ARCH/
> http://ftp.nluug.nl/ftp/pub/os/Linux/distr/fedora/updates/testing/5/$ARCH/
> http://ftp.surfnet.nl/ftp/pub/os/Linux/distr/fedora/5/$ARCH/os/
> http://ftp.nluug.nl/ftp/pub/os/Linux/distr/fedora/5/$ARCH/os/
> http://ftp.surfnet.nl/ftp/pub/os/Linux/distr/fedora/5/$ARCH/os/
> http://ftp.nluug.nl/ftp/pub/os/Linux/distr/fedora/5/$ARCH/os/
>
> That looks suspiciously like the result set for fc5, and fc5 updates (2
> repo's in one response?). However it doesn't look anywhere near like
> ../fedora/core/development/.. repo's :-)
hmm - there does appear to be a bug going on in here somewhere.
I'll take a look. thanks!
Found it. Thanks for pointing it out.
It often helps to make sure that you reinitialize lists rather than just
appending to them w/o bound.... :)
-sv
2:57 p.m.
Hi.
seth vidal <skvidal(a)linux.duke.edu> wrote:
Found it. Thanks for pointing it out.
Well, now it returns just some comments, at least from .de :)
$ wget -q -O -
"http://mirrors.fedoraproject.org/mirrorlist?repo=rawhide&arch=i386"
# no file found for repo = rawhide, country = DE, arch = i386
# filename was: rawhide-DE-i386.txt
$
--
24 hours in a day, 24 beers in a case. Coincidence? I think not.
3:16 p.m.
On Mon, 2006-07-17 at 21:57 +0200, Ralf Ertzinger wrote:
Hi.
seth vidal <skvidal(a)linux.duke.edu> wrote:
> Found it. Thanks for pointing it out.
Well, now it returns just some comments, at least from .de :)
$ wget -q -O -
"http://mirrors.fedoraproject.org/mirrorlist?repo=rawhide&arch=i386"
# no file found for repo = rawhide, country = DE, arch = i386
# filename was: rawhide-DE-i386.txt
ah ha!.
I could only test with global or US from myself. Are you on irc? I could
use some help testing this.
thank you.
-sv
3:56 p.m.
On Mon, 2006-07-17 at 16:16 -0400, seth vidal wrote:
On Mon, 2006-07-17 at 21:57 +0200, Ralf Ertzinger wrote:
> Hi.
>
> seth vidal <skvidal(a)linux.duke.edu> wrote:
>
> > Found it. Thanks for pointing it out.
>
> Well, now it returns just some comments, at least from .de :)
>
> $ wget -q -O -
"http://mirrors.fedoraproject.org/mirrorlist?repo=rawhide&arch=i386"
> # no file found for repo = rawhide, country = DE, arch = i386
> # filename was: rawhide-DE-i386.txt
ah ha!.
I could only test with global or US from myself. Are you on irc? I could
use some help testing this.
okay - I think I have it working more properly now.
Thank you for reporting this.
this is what it will do:
If you do not add &country=DE (or your country code) but there is no
specific set of mirrors that are current for your country then it will
return the global set of current mirrors.
if you do specify a country you want and there is no specific set of
mirrors then it will hand you back an empty file. This is for the people
who don't want to use bandwidth outside of their country for cost
reasons, proxy reasons, etc.
-sv
3:58 p.m.
seth vidal wrote:
On Mon, 2006-07-17 at 16:16 -0400, seth vidal wrote:
> On Mon, 2006-07-17 at 21:57 +0200, Ralf Ertzinger wrote:
>> Hi.
>>
>> seth vidal <skvidal(a)linux.duke.edu> wrote:
>>
>>> Found it. Thanks for pointing it out.
>> Well, now it returns just some comments, at least from .de :)
>>
>> $ wget -q -O -
"http://mirrors.fedoraproject.org/mirrorlist?repo=rawhide&arch=i386"
>> # no file found for repo = rawhide, country = DE, arch = i386
>> # filename was: rawhide-DE-i386.txt
> ah ha!.
>
> I could only test with global or US from myself. Are you on irc? I could
> use some help testing this.
>
okay - I think I have it working more properly now.
Thank you for reporting this.
this is what it will do:
If you do not add &country=DE (or your country code) but there is no
specific set of mirrors that are current for your country then it will
return the global set of current mirrors.
if you do specify a country you want and there is no specific set of
mirrors then it will hand you back an empty file. This is for the people
who don't want to use bandwidth outside of their country for cost
reasons, proxy reasons, etc.
What about users who would prefer their country but would like to fall
back on a global list otherwise?
Rahul
4:11 p.m.
On Tue, 2006-07-18 at 02:28 +0530, Rahul wrote:
> If you do not add &country=DE (or your country code) but
there is no
> specific set of mirrors that are current for your country then it will
> return the global set of current mirrors.
>
> if you do specify a country you want and there is no specific set of
> mirrors then it will hand you back an empty file. This is for the people
> who don't want to use bandwidth outside of their country for cost
> reasons, proxy reasons, etc.
>
What about users who would prefer their country but would like to fall
back on a global list otherwise?
Then don't specify a country in the url.
It will look up your country by your ip and if it has its own mirror
file it will use it - otherwise it uses the global file.
-sv
4:29 p.m.
On man, 2006-07-17 at 17:11 -0400, seth vidal wrote:
On Tue, 2006-07-18 at 02:28 +0530, Rahul wrote:
> > If you do not add &country=DE (or your country code) but there is no
> > specific set of mirrors that are current for your country then it will
> > return the global set of current mirrors.
> >
> > if you do specify a country you want and there is no specific set of
> > mirrors then it will hand you back an empty file. This is for the people
> > who don't want to use bandwidth outside of their country for cost
> > reasons, proxy reasons, etc.
> >
>
> What about users who would prefer their country but would like to fall
> back on a global list otherwise?
Then don't specify a country in the url.
It will look up your country by your ip and if it has its own mirror
file it will use it - otherwise it uses the global file.
Maybe one could add several "virtual" "countries" (without getting
into
politics here).
EU, Asia/Pacific, Middle East, Africa (maybe divided into north/south)
could either be gouped together by mirrorlist itself, or it could accept
country=XX as a list (country=XX,YY,ZZ), so one could make a longer
mirrorlist if neccesary.
For small countries with just a few mirrors, there would be much gain in
having the list extended with either neighbouring countries or at least
contries with known good connectivity.
For 'NO' I know there are very few mirrors, but I also know there would
be no problem falling back to 'SE' or 'DK' - or maybe even 'UK' as
all
of those ar fairly tight connected to us.
I also think it would be a good idea to add the global list to the end
of all the lists (or at least those returning less than say 5 mirrors).
That would ensure that one or two delayed or broken mirrors would not
cause an entire country to be left without updates.
Rgds.
Ola Thoresen
4:32 p.m.
On Mon, 2006-07-17 at 23:29 +0200, Ola Thoresen wrote:
EU, Asia/Pacific, Middle East, Africa (maybe divided into
north/south)
could either be gouped together by mirrorlist itself, or it could accept
country=XX as a list (country=XX,YY,ZZ), so one could make a longer
mirrorlist if neccesary.
mirrorlist as a list might be easiest, I think.
I also think it would be a good idea to add the global list to the
end
of all the lists (or at least those returning less than say 5 mirrors).
That would ensure that one or two delayed or broken mirrors would not
cause an entire country to be left without updates.
but then what about people who want to stay inside their country
boundaries b/c of cost issues.
no, I'd rather let the user specify if they want extra stuff if they've
already said they want something specifically.
-sv
4:41 p.m.
On man, 2006-07-17 at 17:32 -0400, seth vidal wrote:
On Mon, 2006-07-17 at 23:29 +0200, Ola Thoresen wrote:
(...)
> I also think it would be a good idea to add the global list to
the end
> of all the lists (or at least those returning less than say 5 mirrors).
> That would ensure that one or two delayed or broken mirrors would not
> cause an entire country to be left without updates.
but then what about people who want to stay inside their country
boundaries b/c of cost issues.
no, I'd rather let the user specify if they want extra stuff if they've
already said they want something specifically.
Agree.
If it accepts country as a list (and preserves the order) I guess one
could just do something like
country=NO,SE,DK,global
Rgds.
Ola Thoresen
9:58 p.m.
On Mon, 2006-07-17 at 23:41 +0200, Ola Thoresen wrote:
Agree.
If it accepts country as a list (and preserves the order) I guess one
could just do something like
country=NO,SE,DK,global
done and checked in.
http://mirrors.fedoraproject.org/cgi-bin/test.py?repo=core-5&arch=i38...
should do what you'd expect it to.
-sv
1:11 a.m.
On man, 2006-07-17 at 22:58 -0400, seth vidal wrote:
On Mon, 2006-07-17 at 23:41 +0200, Ola Thoresen wrote:
>
> Agree.
> If it accepts country as a list (and preserves the order) I guess one
> could just do something like
>
> country=NO,SE,DK,global
>
done and checked in.
http://mirrors.fedoraproject.org/cgi-bin/test.py?repo=core-5&arch=i38...
should do what you'd expect it to.
Perfect!
And since that list is rather short (currently one single mirror) I
tried to add a ',global' at the end, and that worked as well.
Thanks a lot.
Rgds.
Ola Thoresen
1:13 a.m.
On Tue, 2006-07-18 at 08:11 +0200, Ola Thoresen wrote:
On man, 2006-07-17 at 22:58 -0400, seth vidal wrote:
> On Mon, 2006-07-17 at 23:41 +0200, Ola Thoresen wrote:
>
> >
> > Agree.
> > If it accepts country as a list (and preserves the order) I guess one
> > could just do something like
> >
> > country=NO,SE,DK,global
> >
>
> done and checked in.
>
>
http://mirrors.fedoraproject.org/cgi-bin/test.py?repo=core-5&arch=i38...
>
> should do what you'd expect it to.
Perfect!
And since that list is rather short (currently one single mirror) I
tried to add a ',global' at the end, and that worked as well.
Thanks a lot.
of course - don't use the test.py cgi - use the scriptalias to the
production one I posted earlier.
thanks,
-sv
7:15 p.m.
Andy Burns wrote:
for me, geoip seemingly fails to work out that I'm in the UK, and
I get
country=global settings ...
I just checked in the 2006-07-01 GeoIPCountryWhois.csv file,
it seems it should detect me correctly, my subnet is 84.92.25.120/29
"84.92.0.0","84.93.255.255","1415315456","1415446527","GB","United
Kingdom"
7:17 p.m.
On Tue, 2006-07-18 at 01:15 +0100, Andy Burns wrote:
Andy Burns wrote:
> for me, geoip seemingly fails to work out that I'm in the UK, and I get
> country=global settings ...
I just checked in the 2006-07-01 GeoIPCountryWhois.csv file,
it seems it should detect me correctly, my subnet is 84.92.25.120/29
"84.92.0.0","84.93.255.255","1415315456","1415446527","GB","United
Kingdom"
show me the url you're going to and could you paste the resulting
content out to one of the paste locations?
-sv
7:41 p.m.
seth vidal wrote:
show me the url you're going to
http://mirrors.fedoraproject.org/mirrorlist?repo=rawhide&arch=x86_64
and could you paste the resulting
content out to one of the paste locations?
Given that the output is not exactly huge, here it is ...
# repo = rawhide country = global arch = x86_64
http://download.fedoraproject.org/pub/fedora/linux/core/development/$ARCH...
http://mirror.hiwaay.net/redhat/fedora/linux/core/development/$ARCH/os/
http://ftp.dulug.duke.edu/pub/fedora/linux/core/development/$ARCH/os/
http://mirrors.kernel.org/fedora/core/development/$ARCH/os/
http://sunsite.mff.cuni.cz/pub/fedora/development/$ARCH/os/
I also tried
http://mirrors.fedoraproject.org/mirrorlist?repo=rawhide&arch=x86_64&...
but got
# no file found for repo = rawhide, country = GB, arch = x86_64
# filename was: rawhide-GB-x86_64.txt
similarly for country=UK
7:47 p.m.
On Tue, 2006-07-18 at 01:41 +0100, Andy Burns wrote:
seth vidal wrote:
> show me the url you're going to
http://mirrors.fedoraproject.org/mirrorlist?repo=rawhide&arch=x86_64
> and could you paste the resulting
> content out to one of the paste locations?
Given that the output is not exactly huge, here it is ...
# repo = rawhide country = global arch = x86_64
http://download.fedoraproject.org/pub/fedora/linux/core/development/$ARCH...
http://mirror.hiwaay.net/redhat/fedora/linux/core/development/$ARCH/os/
http://ftp.dulug.duke.edu/pub/fedora/linux/core/development/$ARCH/os/
http://mirrors.kernel.org/fedora/core/development/$ARCH/os/
http://sunsite.mff.cuni.cz/pub/fedora/development/$ARCH/os/
I also tried
http://mirrors.fedoraproject.org/mirrorlist?repo=rawhide&arch=x86_64&...
but got
# no file found for repo = rawhide, country = GB, arch = x86_64
# filename was: rawhide-GB-x86_64.txt
similarly for country=UK
That's completely correct.
there are no UK-specific mirrors in the rawhide mirror list that are
current. so it won't hand any back to you.
-sv
7:59 p.m.
seth vidal wrote:
That's completely correct.
there are no UK-specific mirrors in the rawhide mirror list that are
current. so it won't hand any back to you.
oh fair enough, but how does it "know" if
http://www.mirrorservice.org/sites/fedora.redhat.com/development/x86_64/
is up-to-date or not?
8:08 p.m.
On Tue, 2006-07-18 at 01:59 +0100, Andy Burns wrote:
seth vidal wrote:
> That's completely correct.
>
> there are no UK-specific mirrors in the rawhide mirror list that are
> current. so it won't hand any back to you.
oh fair enough, but how does it "know" if
http://www.mirrorservice.org/sites/fedora.redhat.com/development/x86_64/
is up-to-date or not?
Read the check-mirrors.py script in the cvs link I pointed to earlier.
In short - if it doesn't have the same repomd.xml as the mirror master
does then it's not current.
-sv
8:27 p.m.
On Mon, 2006-07-17 at 21:08 -0400, seth vidal wrote:
On Tue, 2006-07-18 at 01:59 +0100, Andy Burns wrote:
> seth vidal wrote:
>
> > That's completely correct.
> >
> > there are no UK-specific mirrors in the rawhide mirror list that are
> > current. so it won't hand any back to you.
>
> oh fair enough, but how does it "know" if
> http://www.mirrorservice.org/sites/fedora.redhat.com/development/x86_64/
> is up-to-date or not?
Read the check-mirrors.py script in the cvs link I pointed to earlier.
In short - if it doesn't have the same repomd.xml as the mirror master
does then it's not current.
Forgive my complete lack of mirror knowledge, but are mirrors told to
rsync this file after the rest of the repo is updated? Otherwise I
would think you'd still catch mirrors in the middle of a repo update if
not.
josh
8:45 p.m.
On Mon, 2006-07-17 at 20:27 -0500, Josh Boyer wrote:
On Mon, 2006-07-17 at 21:08 -0400, seth vidal wrote:
> On Tue, 2006-07-18 at 01:59 +0100, Andy Burns wrote:
> > seth vidal wrote:
> >
> > > That's completely correct.
> > >
> > > there are no UK-specific mirrors in the rawhide mirror list that are
> > > current. so it won't hand any back to you.
> >
> > oh fair enough, but how does it "know" if
> > http://www.mirrorservice.org/sites/fedora.redhat.com/development/x86_64/
> > is up-to-date or not?
>
>
> Read the check-mirrors.py script in the cvs link I pointed to earlier.
>
> In short - if it doesn't have the same repomd.xml as the mirror master
> does then it's not current.
Forgive my complete lack of mirror knowledge, but are mirrors told to
rsync this file after the rest of the repo is updated? Otherwise I
would think you'd still catch mirrors in the middle of a repo update if
not.
nope. A mirror could be in the middle of a sync. We cannot control HOW
the mirrors sync the data. We can't know. But this keeps users from
being bounced to sites with multiple, out of date metadata.
Additionally, if they're given a whole set of mirrors it means that the
mirrors they get will be 'roughly' more current then never pruning the
mirrors at all.
The only other option we have is checking every file and that will take
WAAAAAY too much time and bandwidth.
I'm open to other suggestions.
-sv
5:04 p.m.
On 7/18/06, Jason L Tibbitts III <tibbs(a)math.uh.edu> wrote:
>>>>> "sv" == seth vidal
<skvidal(a)linux.duke.edu> writes:
sv> I'm open to other suggestions.
Just ask the mirrors to --exclude repomd.xml and then rsync it at the
end? It ups the load on the server, though; twice the directory
traversals.
There is also the --delay-updates option to rsync. It causes rsync to
put all the updates in a temporary directory and moves them into place
at the end. The move is fast which makes the update more atomic.
- Ian
6:56 p.m.
On Tue, 2006-07-18 at 15:04 -0700, Ian Burrell wrote:
On 7/18/06, Jason L Tibbitts III <tibbs(a)math.uh.edu> wrote:
> >>>>> "sv" == seth vidal <skvidal(a)linux.duke.edu>
writes:
>
> sv> I'm open to other suggestions.
>
> Just ask the mirrors to --exclude repomd.xml and then rsync it at the
> end? It ups the load on the server, though; twice the directory
> traversals.
>
There is also the --delay-updates option to rsync. It causes rsync to
put all the updates in a temporary directory and moves them into place
at the end. The move is fast which makes the update more atomic.
it's only fast if it doesn't cross a partition.
-sv
1 p.m.
On 7/18/06, seth vidal <skvidal(a)linux.duke.edu> wrote:
On Tue, 2006-07-18 at 15:04 -0700, Ian Burrell wrote:
>
> There is also the --delay-updates option to rsync. It causes rsync to
> put all the updates in a temporary directory and moves them into place
> at the end. The move is fast which makes the update more atomic.
>
it's only fast if it doesn't cross a partition.
The --delay-updates only uses a rename. By default, it puts the files
in temporary directories (named .~tmp~) under each real directory. It
is possible to specifiy the partial directory with the --partial-dir
but using an absolute directory can fail if it is on a different
partition or the tree crosses a mount point because it can't do a
rename.
- Ian
2:28 p.m.
On 18/07/06, seth vidal <skvidal(a)linux.duke.edu> wrote:
> oh fair enough, but how does it "know" if
> http://www.mirrorservice.org/sites/fedora.redhat.com/development/x86_64/
> is up-to-date or not?
Read the check-mirrors.py script in the cvs link I pointed to earlier.
In short - if it doesn't have the same repomd.xml as the mirror master
does then it's not current.
I would be excepotionally surprised if this mirror is not current - in
the past I have corresponded with the mirror maintainer, and I believe
it syncs more than once a day now, due to reports of the fedora mirror
being out of sync (mainly from me).
Jonathan
7:54 p.m.
seth vidal wrote:
Over the last couple of weeks we've put together a mirror
checking tool
and a cgi that will verify the currency of the mirrorlist.
Just wondering if this is more of a single point of failure for updates?
e.g last weekend chunks of IP space inhabited by *.redhat.com were
unroutable from here, causing me problems with yum/yumex.
Admittedly I probably exacerbated the issue by doing a "yum clean all"
to try to solve the issue, which probably threw away all clues that
yum-plugin-fastestmirror might have had to work around the issue, and
meaning that I couldn't even drop back to using "yum -C"
8:11 p.m.
On Tue, 2006-07-18 at 01:54 +0100, Andy Burns wrote:
seth vidal wrote:
> Over the last couple of weeks we've put together a mirror checking tool
> and a cgi that will verify the currency of the mirrorlist.
Just wondering if this is more of a single point of failure for updates?
e.g last weekend chunks of IP space inhabited by *.redhat.com were
unroutable from here, causing me problems with yum/yumex.
If you'd like to take the check-mirrors script that I pointed to in cvs
and use it to generate your own mirrorlists for your client machine you
may. It has very few requirements and is easy to run.
But this is no more of a single point of failure than any other
mirrorlist we might put together. At some point we have to have a list
of mirrors that are known good. They either have to be compiled by each
and every client or be compiled by a server or a set of servers.
Where would you like to place the magical no-single-points-of-failure
router?
-sv
8:31 p.m.
seth vidal wrote:
Where would you like to place the magical
no-single-points-of-failure
router?
:-)
Use round-robin DNS for mirrors.fedoraproject.org resolving to
geographically dispersed servers?
8:43 p.m.
On Tue, 2006-07-18 at 02:31 +0100, Andy Burns wrote:
seth vidal wrote:
> Where would you like to place the magical no-single-points-of-failure
> router?
:-)
Use round-robin DNS for mirrors.fedoraproject.org resolving to
geographically dispersed servers?
and if one of the servers fail then 1 out every N dns requests fail to
find the host b/c the resolving host is down.
oh - but you say - have the dns tables get updated so if a host goes
down it gets taken out of the search? That's why dns caching (yes,
despite TTLs) sucks.
b/c a lot of people will bitch and moan, nevertheless.
and if you think the above can't happen you're wrong. :)
-sv
4:32 p.m.
On Mon, 2006-07-17 at 21:43 -0400, seth vidal wrote:
On Tue, 2006-07-18 at 02:31 +0100, Andy Burns wrote:
> seth vidal wrote:
>
> > Where would you like to place the magical no-single-points-of-failure
> > router?
>
> :-)
>
> Use round-robin DNS for mirrors.fedoraproject.org resolving to
> geographically dispersed servers?
and if one of the servers fail then 1 out every N dns requests fail to
find the host b/c the resolving host is down.
oh - but you say - have the dns tables get updated so if a host goes
down it gets taken out of the search? That's why dns caching (yes,
despite TTLs) sucks.
If the server has multiple DNS aliases and then the client can be
written to fetch multiple aliases in a single DNS request and try
several before giving up.
The client could also cache a copy of the mirror list to fall back. I
imagine the client could even try a couple of servers from the previous
cached list before contacting the central server for a refresh (which
could reduce the load on the server hosting the mirror list).
The even better solution is to have multiple dispersed servers with the
_same_ IP. This can be done for UDP using IP Anycast but isn't suited to
TCP/HTTP traffic since sequential packets can get sent to different
servers.
http://www.net.cmu.edu/pres/anycast/
http://en.wikipedia.org/wiki/Anycast
Jon
5:18 p.m.
New subject: mirrorlist cgi - URLs not ending in country code ?
seth vidal wrote:
Hi,
Over the last couple of weeks we've put together a mirror checking tool
and a cgi that will verify the currency of the mirrorlist. Then it will
generate lists per-country based on geoip country codes as well as a
global mirror list.
The code for this tool is available here:
http://cvs.fedora.redhat.com/viewcvs/check-mirrors/?root=fedora
Cool; for a test, I
tried the test URL with country=AU, and the return
list included a single .au mirror. This leads me to the question of is
it just the list of mirrors' mirror URLs that is being checked for a
country code at the right hand end ?
If so what would need to happen to make for eg:
http://planetmirror.com/blahblah
be returned in the AU list (since it is located in Brisbane Australia) ?
DaveT.
6:59 p.m.
New subject: mirrorlist cgi - URLs not ending in country code ?
On Wed, 2006-07-19 at 08:18 +1000, David Timms wrote:
seth vidal wrote:
> Hi,
> Over the last couple of weeks we've put together a mirror checking tool
> and a cgi that will verify the currency of the mirrorlist. Then it will
> generate lists per-country based on geoip country codes as well as a
> global mirror list.
>
> The code for this tool is available here:
>
> http://cvs.fedora.redhat.com/viewcvs/check-mirrors/?root=fedora
Cool; for a test, I tried the test URL with country=AU, and the return
list included a single .au mirror. This leads me to the question of is
it just the list of mirrors' mirror URLs that is being checked for a
country code at the right hand end ?
If so what would need to happen to make for eg:
http://planetmirror.com/blahblah
be returned in the AU list (since it is located in Brisbane Australia) ?
As I said at the beginning of the thread it's using geoip.
we take the ip of the host in the mirror and hand that to geoip which
then tells us which country an ip lives in.
-sv
9:07 a.m.
New subject: mirrorlist cgi - URLs not ending in country code ?
seth vidal wrote:
On Wed, 2006-07-19 at 08:18 +1000, David Timms wrote:
> seth vidal wrote:
>> Hi,
>> Over the last couple of weeks we've put together a mirror checking tool
...
As I said at the beginning of the thread it's using geoip.
we take the ip of the host in the mirror and hand that to geoip which
then tells us which country an ip lives in.
OK, thanks. I guess the mirrorlist cgi
would use the GeoLite DB
{http://www.maxmind.com/app/geoip_country} which suggests:
Accuracy Approximately 97%. I could imagine that there would be a large
amount of changes per day, and hence impossible to keep it truely up2date.
DaveT.
9:17 a.m.
New subject: mirrorlist cgi - URLs not ending in country code ?
On Thu, 2006-07-20 at 00:07 +1000, David Timms wrote:
seth vidal wrote:
> On Wed, 2006-07-19 at 08:18 +1000, David Timms wrote:
>> seth vidal wrote:
>>> Hi,
>>> Over the last couple of weeks we've put together a mirror checking
tool
...
> As I said at the beginning of the thread it's using geoip.
>
> we take the ip of the host in the mirror and hand that to geoip which
> then tells us which country an ip lives in.
OK, thanks. I guess the mirrorlist cgi would use the GeoLite DB
{http://www.maxmind.com/app/geoip_country} which suggests:
Accuracy Approximately 97%. I could imagine that there would be a large
amount of changes per day, and hence impossible to keep it truely up2date.
which is significantly better than doing nothing at all.
perfection runs contrary to beneficial.
-sv
5:49 p.m.
New subject: mirrorlist cgi - legit repo= values?
seth vidal wrote:
Hi,
Over the last couple of weeks we've put together a mirror checking tool
and a cgi that will verify the currency of the mirrorlist. Then it will
generate lists per-country based on geoip country codes as well as a
global mirror list.
...
[development]
name=Fedora Core - Development
mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=rawhide&a...
and for extras:
[extras-development]
name=Fedora Extras - Development Tree
mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=extras-devel&...
I have tried this on an FC5 + nearly up2date machine, and with the
"correct" repo=, it produced a useful mirrorlist and went ahead and
updated both core and extras OK.
##mirrorlist=http://fedora.redhat.com/download/mirrors/updates-released-fc$releasever
mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=core-$release...
##mirrorlist=http://fedora.redhat.com/download/mirrors/fedora-extras-$releasever
mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=extras-$relea...
Would it make sense for the repo= to be the repo name, and encode the
version eg 1..whatever + rawhide as version=$releasever ?
ie:
mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=extras&ve...
Also when yum reads the .repo, could it use the name from either x.repo
or the repos defined internally as [extras-debuginfo] etc as a $variable ?
David Timms.
6:58 p.m.
New subject: mirrorlist cgi - legit repo= values?
On Wed, 2006-07-19 at 08:49 +1000, David Timms wrote:
seth vidal wrote:
> Hi,
> Over the last couple of weeks we've put together a mirror checking tool
> and a cgi that will verify the currency of the mirrorlist. Then it will
> generate lists per-country based on geoip country codes as well as a
> global mirror list.
...
> [development]
> name=Fedora Core - Development
>
mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=rawhide&a...
>
>
> and for extras:
>
> [extras-development]
> name=Fedora Extras - Development Tree
>
mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=extras-devel&...
I have tried this on an FC5 + nearly up2date machine, and with the
"correct" repo=, it produced a useful mirrorlist and went ahead and
updated both core and extras OK.
##mirrorlist=http://fedora.redhat.com/download/mirrors/updates-released-fc$releasever
mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=core-$release...
##mirrorlist=http://fedora.redhat.com/download/mirrors/fedora-extras-$releasever
mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=extras-$relea...
Would it make sense for the repo= to be the repo name, and encode the
version eg 1..whatever + rawhide as version=$releasever ?
ie:
mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=extras&ve...
Also when yum reads the .repo, could it use the name from either x.repo
or the repos defined internally as [extras-debuginfo] etc as a $variable ?
why bother adding all that complexity to yum?
it's just a url - how it is comprised makes no difference at all.
-sv
9:20 a.m.
New subject: mirrorlist cgi - legit repo= values?
seth vidal wrote:
On Wed, 2006-07-19 at 08:49 +1000, David Timms wrote:
...
why bother adding all that complexity to yum?
With out a clear
understanding of yum, I imagined that such other
variables might already exist.
it's just a url - how it is comprised makes no difference at all.
Yes, on second thought I agree that kiss is the way to go. I was
thinking about the differences between a repo file between fedora
versions, and developers not having to change the .repo files as
development/releases continue. But the $releasever really takes care of
that for the most part already {sans devel?}.
However, what defines the legitimate repo=X values ? Is this specified
elsewhere, or made up and hand crafted into release .repo files. Perhaps
a non-existent repo=whatgoeshere request could provide the list of
legit values (perhaps as comments) in the returned mirrorlist ?
DaveT.
9:19 a.m.
New subject: mirrorlist cgi - legit repo= values?
On Thu, 2006-07-20 at 00:20 +1000, David Timms wrote:
seth vidal wrote:
> On Wed, 2006-07-19 at 08:49 +1000, David Timms wrote:
...
> why bother adding all that complexity to yum?
With out a clear understanding of yum, I imagined that such other
variables might already exist.
> it's just a url - how it is comprised makes no difference at all.
Yes, on second thought I agree that kiss is the way to go. I was
thinking about the differences between a repo file between fedora
versions, and developers not having to change the .repo files as
development/releases continue. But the $releasever really takes care of
that for the most part already {sans devel?}.
However, what defines the legitimate repo=X values ? Is this specified
elsewhere, or made up and hand crafted into release .repo files. Perhaps
a non-existent repo=whatgoeshere request could provide the list of
legit values (perhaps as comments) in the returned mirrorlist ?
go to mirrors.fedoraproject.org
look at the file names you see there.
the format of the filenames is:
repo-countrycode-arch.txt
-sv
1:29 p.m.
On Monday 17 July 2006 12:13, seth vidal wrote:
[development]
name=Fedora Core - Development
mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=rawhide&a...
asearch
and for extras:
[extras-development]
name=Fedora Extras - Development Tree
mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=extras-devel&...
ch=$basearch
I've added these to CVS for fedora-release. The fedora-release package I spin
up prior to Test2 (for use in test2) should have it included.
--
Jesse Keating
Release Engineer: Fedora
6491
days inactive
6497
days old
42 comments
13 participants
participants (13)
-
Andy Burns -
Chris Chabot -
David Timms -
Ian Burrell -
Jason L Tibbitts III -
Jesse Keating -
Jon Burgess -
Jonathan Underwood -
Josh Boyer -
Ola Thoresen -
Rahul Sundaram -
Ralf Ertzinger -
seth vidal