• Registered: 2008-11-28
  • Posts: 36

Topic: googlebot and punbb

I'm not 100% sure but I think googlebot can read parts of the forums that are restricted from guests sad

I have htaccess to disallow googlebot so google webmaster tools is listing some private threads as URL restricted by robots.txt
If googlebot can't see them as a guest, how is it trying to crawl them to begin with? or does punbb let googlebot see private forum sections?

  • Slavok
  • PunBB Developer
  • Offline
  • From: Russia
  • Registered: 2008-06-11
  • Posts: 1,238

Re: googlebot and punbb

hussam wrote:

If googlebot can't see them as a guest, how is it trying to crawl them to begin with? or does punbb let googlebot see private forum sections?

No, if a forum is private, googlebot can't crawl it. Could you post here (or send me a pm with) the links or something what can confirm your thoughts?

Slavok's Website

  • Registered: 2008-11-28
  • Posts: 36

Re: googlebot and punbb

PM sent. thank you very much.

  • Registered: 2008-11-28
  • Posts: 36

Re: googlebot and punbb

I think I will remove the robots.txt entry and see what it indexes.

  • Registered: 2010-03-09
  • Posts: 7

Re: googlebot and punbb

Googlebot = Guest. Though some forum platforms show Search Engine bots as Registered User. I wonder if it actually gives access to member-only pages and contents.

Nathan Anderson: Life Quotes / Law of Attraction
  • Slavok
  • PunBB Developer
  • Offline
  • From: Russia
  • Registered: 2008-06-11
  • Posts: 1,238

Re: googlebot and punbb

Nathan Anderson wrote:

Googlebot = Guest. Though some forum platforms show Search Engine bots as Registered User. I wonder if it actually gives access to member-only pages and contents.

No, it will not open private forums for bots.

Slavok's Website