Our crawlers should both connect natted from 209.132.181.102
Adrian
On Mon, Oct 09, 2017 at 10:31:30AM -0400, Mike Neir wrote:
Greetings Adrian,
I'll follow up with my team and see if we can get to the bottom of this
issue. As far as I know, there shouldn't be any CDN involvement with our
mirror, just content served from one machine. Can you tell provide
information on which IP address(es) those queries might be coming from so
we can search more efficiently?
MN
Mike Neir
Infrastructure Administrator
Liquid Web, Inc.
On Mon, Oct 9, 2017 at 8:42 AM, Adrian Reber <adrian(a)lisas.de> wrote:
> Hello
mirrors.liquidweb.com admins,
>
> Fedora's mirror crawler has trouble crawling your mirror as it does
> return a unusual status code:
>
> # curl
http://mirrors.liquidweb.com/fedora-epel/ -v
> * About to connect() to
mirrors.liquidweb.com port 80 (#0)
> * Trying 69.167.187.144...
> * Connected to
mirrors.liquidweb.com (69.167.187.144) port 80 (#0)
> > GET /fedora-epel/ HTTP/1.1
> > User-Agent: curl/7.29.0
> > Host:
mirrors.liquidweb.com
> > Accept: */*
> >
> < HTTP/1.1 000
> < Server: nginx
> < Date: Mon, 09 Oct 2017 12:37:13 GMT
> < Content-Type: text/html
> < Transfer-Encoding: chunked
> < Connection: keep-alive
>
>
> The crawler expects the status code 200 if everything works correctly and
> ignores your mirror because of the status code 000. Any reason for
> status code 000. It seems that the code 000 is something Akamai uses for
> requests where the client aborted:
>
> """
> Log Delivery Services (LDS) LDS will show a 000 for any 200 or 206
> responses with a client abort: the object was served correctly from the
> origin or edge, but the end-user terminated the connection/transaction
> before it completed.
> """
>
> Not sure if that is related to your situation. But right now your mirror
> is disabled as it seems to be inaccessible (at least from the crawlers
> point of view).
>
> Adrian
>