• Registered: 2008-10-20
  • Posts: 41

Topic: robots.txt

With SEO in mind, which PunBB 1.2 files should I disallow with robots.txt? I noticed that one of the admin files has already been indexed by Yahoo. And I'm sure we don't want that. Should I disallow everything but but the index file?

Thanks.

  • sirena
  • sirena
  • Senior Member
  • Offline
  • Registered: 2006-04-06
  • Posts: 473

Re: robots.txt

No. All you should really disallow is register.php.

  • Registered: 2008-10-20
  • Posts: 41

Re: robots.txt

Why only register.php? That's where I added captcha to avoid spam, but the idea of robots.txt is to disallow files without real content so the robots don't waste time crawling them. Like the irrelevant admin files which have already been indexed by Yahoo.

  • sirena
  • sirena
  • Senior Member
  • Offline
  • Registered: 2006-04-06
  • Posts: 473

Re: robots.txt

AlexanderS wrote:

Why only register.php? That's where I added captcha to avoid spam, but the idea of robots.txt is to disallow files without real content so the robots don't waste time crawling them. Like the irrelevant admin files which have already been indexed by Yahoo.

So then just disallow all the admin files then. It's not a big deal, since most of them shouldn't run if directly called anyhow, I think.
Disallowing register.php is helpful even if you have a captcha in place. All captchas and registration page tricks have potential flaws or weaknesses.  Disallowing register.php means bots who mine Google etc to locate the registration pages of PunBB forums (which is how some of them work) won't show a result for your site.