We've been seeing load spikes on hostedXX, following df7e8578432b224d9576dc8359f0729763861526. This semi-reverts that commit and only allows /wiki/* to be crawled.
diff --git a/configs/web/fedorahosted.org/fedorahosted-robots.txt b/configs/web/fedorahosted.org/fedorahosted-robots.txt index cd572f8..7782677 100644 --- a/configs/web/fedorahosted.org/fedorahosted-robots.txt +++ b/configs/web/fedorahosted.org/fedorahosted-robots.txt @@ -1,5 +1,5 @@ User-agent: * -Disallow: /*/browser -Disallow: /*/search +Allow: /wiki/* +Disallow: / user-agent: AhrefsBot disallow: /
On Sun, 30 Dec 2012 20:07:37 -0500 Ricky Elrod codeblock@elrod.me wrote:
We've been seeing load spikes on hostedXX, following df7e8578432b224d9576dc8359f0729763861526. This semi-reverts that commit and only allows /wiki/* to be crawled.
diff --git a/configs/web/fedorahosted.org/fedorahosted-robots.txt b/configs/web/fedorahosted.org/fedorahosted-robots.txt index cd572f8..7782677 100644 --- a/configs/web/fedorahosted.org/fedorahosted-robots.txt +++ b/configs/web/fedorahosted.org/fedorahosted-robots.txt @@ -1,5 +1,5 @@ User-agent: * -Disallow: /*/browser -Disallow: /*/search +Allow: /wiki/* +Disallow: / user-agent: AhrefsBot disallow: /
It seems like some things are causing the load, but it's hard to isolate (in particular timeline, changeset and log since they all hit the repo browser).
I'm +1 to just applying this one for now and we can adjust it further after the freeze is over.
kevin
On 30 December 2012 18:12, Kevin Fenzi kevin@scrye.com wrote:
On Sun, 30 Dec 2012 20:07:37 -0500 Ricky Elrod codeblock@elrod.me wrote:
We've been seeing load spikes on hostedXX, following df7e8578432b224d9576dc8359f0729763861526. This semi-reverts that commit and only allows /wiki/* to be crawled.
diff --git a/configs/web/fedorahosted.org/fedorahosted-robots.txt b/configs/web/fedorahosted.org/fedorahosted-robots.txt index cd572f8..7782677 100644 --- a/configs/web/fedorahosted.org/fedorahosted-robots.txt +++ b/configs/web/fedorahosted.org/fedorahosted-robots.txt @@ -1,5 +1,5 @@ User-agent: * -Disallow: /*/browser -Disallow: /*/search +Allow: /wiki/* +Disallow: / user-agent: AhrefsBot disallow: /
+1 Also.
infrastructure@lists.fedoraproject.org