Topic: robots.txt

With SEO in mind, which PunBB 1.2 files should I disallow with robots.txt? I noticed that one of the admin files has already been indexed by Yahoo. And I'm sure we don't want that. Should I disallow everything but but the index file?

Thanks.

Re: robots.txt

No. All you should really disallow is register.php.

Re: robots.txt

Why only register.php? That's where I added captcha to avoid spam, but the idea of robots.txt is to disallow files without real content so the robots don't waste time crawling them. Like the irrelevant admin files which have already been indexed by Yahoo.

Re: robots.txt

AlexanderS wrote:

Why only register.php? That's where I added captcha to avoid spam, but the idea of robots.txt is to disallow files without real content so the robots don't waste time crawling them. Like the irrelevant admin files which have already been indexed by Yahoo.

So then just disallow all the admin files then. It's not a big deal, since most of them shouldn't run if directly called anyhow, I think.
Disallowing register.php is helpful even if you have a captcha in place. All captchas and registration page tricks have potential flaws or weaknesses.  Disallowing register.php means bots who mine Google etc to locate the registration pages of PunBB forums (which is how some of them work) won't show a result for your site.