While looking at bugs assigned to the Docs team a saw at least 3 bugs that talked about Publican being a temp solution. It seems that at least a few people outside of Docs are not that happy with what Publican provides.

-----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/ iQIcBAEBAgAGBQJMCC7YAAoJEDbiLlqcYamxGA8QAJUqqofCeoL7QkfsorTy8Nhh gKBs06HH7SfQ2eaIm3MDvs4HrnIUlQL9f5hJ89jGis4eKi2FBvistz775cRXWqSe SRDDP1nMCwqzVPz8nhc1dsd6fEpXkuuuhXKrnHajW6rhJx+r4HY592vditM71gUk UVEkO88D1ni8ixyXfo7XWK6X16ucFJ74VY0c7hR+SG16ve5y6UD361nwICfoQHYF ACRCsgvQP6j9OExanoOqvZd1ko0XrbxGuA/myfrrU9BL1sFPJwWUtkyN3xF4JVdN VeyHfVbgGsPXJykTNe12eI7teTIXxmF1IzE2eYVDNsOA/t8exatd7Jdo1yxE6edM 8ouvuNDKFMHOQk4cGqj/mb+4Ud7BmckY1HIaKVqOuLj4gb8OlknZ/CW/OmViETvT 6c7j4k2lkqXjVZYCNEMsMmvuRyDntN+HDNQRLa7aTkgybc/38DE+UDhtWWAK1PiJ ts3993zJTk8VeMWxQlrt1PFKeBWXvg+GirzbaaxlRdNUBYyvKWQI7/zoG2k1jX1x TLviVKD0swKLa2lHeD6VPv9qJ09G23fpQYnQ3wzszjfzijHylNEynBZJUAE2lFev 7rBdI0POIgEvs+kc4LumnrS8gOZR2kwZEwYZLBK3lC7MDraXh7ZjBXLvSHBkGKp3 vtdFf0QAONGnpmd3megb =bPbI -----END PGP SIGNATURE-----

I don't consider myself "outside" but the most glaring problem we have now is what has been induced by changing the entire site structure: * Broken links from Google that bring the user back to a front page * An embedded search that brings users to Google, which doesn't work and often brings the user back... (etc.) Some of this might change as Google re-indexes content, but we really do need to keep a careful eye on how the user experience of finding our docs content is working.

-----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/ iQIcBAEBAgAGBQJMCTfzAAoJEDbiLlqcYamxAm4P/jv/lW21uAB/WHd4mmk2YI36 kQsNvFFzgxKrzE2x5qZsbp2TVUDHrzHOHoYENR4pa5DiMpmSvM7dCTAvgLj5Auoe HiFTj5w3bYgX1Kf9J0VY/bbFRJq0FhAlF7l5ZVSbCu5NXiIKxOPw5vbHeOPvMphY VS++zYvQ8wIZTQ0E1atawJcLZsooRRHlhJv7/UgoLuKuZjKAAjWFjhrEvIQxSAzY HFk74CY0Qwr5/b4p+eaotnecGYbjTeE8T0uY3iqkQKEa0hyFwUTIhZ0WQ8y4Vh3x x7LauINcdDFOLPOHmQX26g6n597uhijed+r9K0yBNJkLDX8afDmCYdcpVU3hZ7hj 64GRx3yWKta7+o1uZD+lnv3U52B69H2mSc7ihZApBBkmBuNX/+Mi+uLBvXdSW120 GiKX3OA+nluQpUl189YbGZ+eJ47l9ZTN6Rp48i1su/1gcRGh1KmTl72WJpOXSNZR U9qxrwKB5LONTTCg5hqp/Zqe54g7udtrYEWucSMWtc5ImaIQ5RSY35+FtXchgD+0 lfduDB1Pzob0qEBRIksrMoMrouitdGlar3myK0sI1Oljw5ns6fO3Es9GgNaeJpsD FjMNPE7N2Yct6bw5RpOy1cVzKc9hyqDC157F2d74Q2ritMhBq6psqL/bteiO+nbR xc2ZFzAW1CP4TrxtVxeV =9v3x -----END PGP SIGNATURE-----

On Fri, Jun 04, 2010 at 13:29:24 -0400, Eric Sparks Christensen <sparks(a)fedoraproject.org&gt; wrote: > > Yeah, that was a problem. It would be nice to know how often Google > indexes the site, when the last changes could be made to the structure > before a release, and if we could force a re-index. Some of this is described here: http://www.googleguide.com/google_works.html It looks like you could get some new stuff added using: http://www.google.com/addurl/?continue=/addurl That isn't the same as reindexing existing stuff though. And I don't know if readding a link Google knows about triggers a reindex.

One of the new features of Publican 2.0 that I haven't mentioned yet is that it creates an XML sitemap for search engine bots to crawl. You can find d.fp.o.'s sitemap here: http://docs.fedoraproject.org/Sitemap

I've fed this to Google, Yahoo, and Bing, and they're all slowly re-indexing the site. The map now contains a little over 2,000 URLs and at the time of writing, Google has crawled about 350 of them.

The dilemma we face is the decision of when to turn off the 404 redirect. For the sake of all the existing links scattered around the net (both on the Fedora Project site and off it), we'd want to postpone this as far as possible. On the other hand, any bot attempting to verify that link gets a page served up and probably concludes that the link is valid; I suspect that if these links 404ed, they'd start to evaporate from search results.

Given that existing links around the net are pointing to (at most recent) the F12 versions of docs, there will be no need to keep the 404 redirect in place past October; however, if we want to start allowing dead links to 404 out rather than poison search results, maybe we should bring that date forward? The sooner we do this, the sooner search will start working properly...

Cheers Rudi

-----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/ iQIcBAEBAgAGBQJMCh74AAoJEDbiLlqcYamx570P/igKV+7uS9AlIYWIYbSgTANw scl/oVSzcIvj3A6O9ddFgPZLBg7fHkXXCh8TtExttDC71Xl+nromHQH4B07H35+g /35qnojQF1RZhDcBhzYbvlCbT5wNvJZ3Tg1/9yGixfgArPuwfB5UlDKWHhbPeMBF 15Hz2vEd140NCDcHZphZy8CZgvmFD655GfkM6bZ3J0CXfFTxgv20qI7AdjT+H/Fu D31VZxnxNsIB4TwozgWkzAdyFRQ5DVJjsrYihUt+D3RVDXZ9U9eFaTU7gtNv17sz bpkSati7R6pJ40wpHG5tYnlLSaSEmk0qIkyzopl90ZQ80WS7aeS9fdHZGvk9YZqY YFk3fY/pH7KGzSIxN48y1Ai3RSR2XGsBoM70TVsHzaL498UY2gu8NzgsfLKsz5Qo 4NQJQJhm3+t2G4ONuAlA2l6fglppyRqp65lJNPTp/iNPnx+y9VcwL7hcZL8wUSE+ jZfSgVz04xXlxrK04wfrTOcSOxRmf678fMdQSLuOG/WXgrWlZ8gaAwa+z8vcCo/U H/xab3DqQ2zV562fkRnj5ALhksbz0hW6ieK1y7cAMCR2CK6o0ZyFCjfQS3OFF6ZC rVimBtAHrkM6mQEJutzcUeIJk/H5iFT9hol06YQ64Ftq4SZX47IKCit8CSBKgvvb yivlbYiORTmnfv69cd4U =B189 -----END PGP SIGNATURE-----

Including Infrastructure team gurus on this email thread for additional expert advice and assistance. :-) Setting reply-to docs@ list. On Sat, Jun 05, 2010 at 05:55:05AM -0400, Eric Sparks Christensen wrote: > On 06/05/2010 03:14 AM, Ruediger Landmann wrote: > > One of the new features of Publican 2.0 that I haven't mentioned yet is > > that it creates an XML sitemap for search engine bots to crawl. You can > > find d.fp.o.'s sitemap here: > > > > http://docs.fedoraproject.org/Sitemap > > Awesome. > > > > > I've fed this to Google, Yahoo, and Bing, and they're all slowly > > re-indexing the site. The map now contains a little over 2,000 URLs and > > at the time of writing, Google has crawled about 350 of them. > > I know that Google has some algorithm that figures out how often your > site changes and then crawls more or less frequently.  Not sure if we > could work with Google on scheduling this more or less around the time > of a release.  Of course I'm guessing that we won't have this big > re-structuring next time, either. > > > > > The dilemma we face is the decision of when to turn off the 404 > > redirect. For the sake of all the existing links scattered around the > > net (both on the Fedora Project site and off it), we'd want to postpone > > this as far as possible. On the other hand, any bot attempting to verify > > that link gets a page served up and probably concludes that the link is > > valid; I suspect that if these links 404ed, they'd start to evaporate > > from search results. > > Isn't there a type of redirect (302?) that tells you that you are being > redirected so you don't think the URL is valid? I thought that 301 and 302 both do this, but 302 generally is used for temporary redirects, and 301 for permanent ones.  So 301 would be the one you're thinking of here perhaps?

> > Given that existing links around the net are pointing to (at most > > recent) the F12 versions of docs, there will be no need to keep the 404 > > redirect in place past October; however, if we want to start allowing > > dead links to 404 out rather than poison search results, maybe we should > > bring that date forward? The sooner we do this, the sooner search will > > start working properly... > > Yeah, the sooner the better, IMO. I wonder how long you would need 301's in place before it's safe to remove them?  Because I think Infrastructure's not keen on maintaining a big list of these.

-- Paul W. Frields                                http://paul.frields.org/  gpg fingerprint: 3DA6 A0AC 6D58 FEC4 0233  5906 ACDB C937 BD11 3717  http://redhat.com/   -  -  -  -   http://pfrields.fedorapeople.org/          Where open source multiplies: http://opensource.com _______________________________________________ infrastructure mailing list infrastructure(a)lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/infrastructure

The dilemma we face is the decision of when to turn off the 404 redirect. For the sake of all the existing links scattered around the net (both on the Fedora Project site and off it), we'd want to postpone this as far as possible. On the other hand, any bot attempting to verify that link gets a page served up and probably concludes that the link is valid; I suspect that if these links 404ed, they'd start to evaporate from search results.