• From: 127.0.0.1
  • Registered: 2001-11-02
  • Posts: 9,167

Topic: Preventing search bot indexing of certain pages

There are some pages in a bulletin board that you don't want search engines to index. Some examples are: post.php, login.php, search.php etc etc. You basically want search engines to only index the pages that have real content, e.g. index.php, viewforum.php, viewtopic.php and profile.php. A solution to the problem is to add a meta element to the pages that we don't want indexed? Something like this:

<meta name="ROBOTS" content="NOINDEX" />

How does that sound?

"Programming is like sex: one mistake and you have to support it for the rest of your life."

Rickard's Website

  • From: Gothenburg, Sweden
  • Registered: 2002-07-03
  • Posts: 946

Re: Preventing search bot indexing of certain pages

Sounds like a good idea.

Are there any drawbacks using this method?

  • From: 127.0.0.1
  • Registered: 2001-11-02
  • Posts: 9,167

Re: Preventing search bot indexing of certain pages

That's why I asked. I have no idea :)

Edit: Here's some reading.

"Programming is like sex: one mistake and you have to support it for the rest of your life."

Rickard's Website

4 Reply by hcgtv

  • hcgtv
  • hcgtv
  • Senior Member
  • Offline
  • From: Charlotte, NC
  • Registered: 2004-06-25
  • Posts: 1,991

Re: Preventing search bot indexing of certain pages

Couldn't you use a robots.txt file?

PHPCrossRef . TXP Make . TXP Themes . TXP Tags . TXP Planet . We Love TXP

hcgtv's Website

  • From: 127.0.0.1
  • Registered: 2001-11-02
  • Posts: 9,167

Re: Preventing search bot indexing of certain pages

A robots.txt is an alternative. I just thought the meta tag thing was a bit more elegant. Maybe not.

"Programming is like sex: one mistake and you have to support it for the rest of your life."

Rickard's Website

  • Registered: 2004-08-16
  • Posts: 577

Re: Preventing search bot indexing of certain pages

hmm... ill try it when i get my site on google. ive never had a site in a search engine before, so i have almost no clue how this stuff works sad

Indocron
$theQuestion = (2*b) || !(2*b);
  • Registered: 2004-08-08
  • Posts: 508

Re: Preventing search bot indexing of certain pages

Gary, visit robotstxt.org for more info

  • From: -8 GMT
  • Registered: 2004-05-04
  • Posts: 1,396

Re: Preventing search bot indexing of certain pages

i would suggest to EXCLUDE the profile.php also so as to not allow bots to collect emails also.  just a suggestion.

~James
FluxBB - Less is more

9 Reply by snapsolutions (edited by snapsolutions 2004-09-22 00:42)

  • Registered: 2004-08-08
  • Posts: 508

Re: Preventing search bot indexing of certain pages

robots can't fetch your email address from the profile menu

  • From: 127.0.0.1
  • Registered: 2001-11-02
  • Posts: 9,167

Re: Preventing search bot indexing of certain pages

Unless you explicitly choose to have your e-mail address publicly visible.

"Programming is like sex: one mistake and you have to support it for the rest of your life."

Rickard's Website

  • Connorhd
  • Connorhd
  • Former PunBB Developer
  • Offline
  • From: Leeds, UK
  • Registered: 2004-06-13
  • Posts: 4,195

Re: Preventing search bot indexing of certain pages

any i doubt a meta-tag stops email harvestors

FluxBB.
Connorhd.co.uk.

Connorhd's Website

  • Frank H
  • Former PunBB Developer
  • Offline
  • Registered: 2003-03-16
  • Posts: 1,423

Re: Preventing search bot indexing of certain pages

here's a good explanation how to write robots.txt files
http://www.robotstxt.org/wc/norobots.html

  • From: 127.0.0.1
  • Registered: 2001-11-02
  • Posts: 9,167

Re: Preventing search bot indexing of certain pages

There's a downside to using a robots.txt, it must be accessible via the URL "/robots.txt". That means that it will have be placed in the domain or subdomain root directory even if the forums are accessed via e.g. "/forums/". This in turn means that robots.txt will have to be edited depending on in which directory the forums are located.

"Programming is like sex: one mistake and you have to support it for the rest of your life."

Rickard's Website

  • Frank H
  • Former PunBB Developer
  • Offline
  • Registered: 2003-03-16
  • Posts: 1,423

Re: Preventing search bot indexing of certain pages

yep ... unfortunately, but at least many robots search for it, so I guess they use it aswell wink

  • Connorhd
  • Connorhd
  • Former PunBB Developer
  • Offline
  • From: Leeds, UK
  • Registered: 2004-06-13
  • Posts: 4,195

Re: Preventing search bot indexing of certain pages

1 question.. why would you want to stop robots scanning your pages? surely the more pages the better, even if they are not very useful

FluxBB.
Connorhd.co.uk.

Connorhd's Website

  • From: 127.0.0.1
  • Registered: 2001-11-02
  • Posts: 9,167

Re: Preventing search bot indexing of certain pages

I hate searching for something and finding the response in e.g. a post form instead of the actual topic.

"Programming is like sex: one mistake and you have to support it for the rest of your life."

Rickard's Website

17 Reply by lorax

  • Registered: 2004-04-14
  • Posts: 35

Re: Preventing search bot indexing of certain pages

For the bots that DO follow the rules you should use:

<meta name="ROBOTS" content="NOINDEX,FOLLOW" />

which will tell the bots not to index the current page but to follow the links on the page to other pages.

For those bots that don't play nice I ban them using htaccess.

Re: why? I don't need my bandwidth being eaten up by a rogue bot, email harvester, website whacker, or some wannabe hacker chewing through the pages on my forum.

18 Reply by zaher

  • zaher
  • Senior Member
  • Offline
  • From: Syria, Damascus
  • Registered: 2004-02-16
  • Posts: 773

Re: Preventing search bot indexing of certain pages

Some one found my site by search for a Joke in google,  that joke found in my forum in "General discussions" forum, i hate that 

I think it is good to execlude "General discussions" in that way, maybe add option to forum options.

If your people come crazy, you will not need to your mind any more.
  • From: 127.0.0.1
  • Registered: 2001-11-02
  • Posts: 9,167

Re: Preventing search bot indexing of certain pages

lorax: Good idea.

zaher: That, on the other hand, will require a forum specific option and I'm not sure it's worth it.

"Programming is like sex: one mistake and you have to support it for the rest of your life."

Rickard's Website

  • Connorhd
  • Connorhd
  • Former PunBB Developer
  • Offline
  • From: Leeds, UK
  • Registered: 2004-06-13
  • Posts: 4,195

Re: Preventing search bot indexing of certain pages

zaher thats a bit bad because your genral discussions might just happen to have the joke they are looking for wink

FluxBB.
Connorhd.co.uk.

Connorhd's Website

  • From: India
  • Registered: 2004-06-03
  • Posts: 435

Re: Preventing search bot indexing of certain pages

If we remove the meta tag of keyword and description from our pages, still the search engine will spot it or excluively we have to define like <meta name="ROBOTS" content="NOINDEX" />

God wisely designed the human body so that we can neither pat our own backs nor kick ourselves too easily
  • From: 127.0.0.1
  • Registered: 2001-11-02
  • Posts: 9,167

Re: Preventing search bot indexing of certain pages

This has now been implemented in 1.2. The meta tag

<meta name="ROBOTS" content="NOINDEX, FOLLOW" />

can be found on alla pages except: index.php, viewforum.php, viewtopic.php, profile.php (viewing a profile as a guest) and userlist.php. Apparently, not all robots support the meta tag, but the important ones do (i.e. Googlebot :D).

"Programming is like sex: one mistake and you have to support it for the rest of your life."

Rickard's Website

23 Reply by nrwilk (edited by nrwilk 2006-04-15 17:06)

  • Registered: 2006-04-15
  • Posts: 14

Re: Preventing search bot indexing of certain pages

May I ask how I can block search bots from scanning ANY forum pages?

Should I just add 

<meta name="ROBOTS" content="NOINDEX, NOFOLLOW" />

to the top of all pages associated with PunnBB?

Thanks!

  • From: Leuven, Belgium
  • Registered: 2005-06-14
  • Posts: 2,653

Re: Preventing search bot indexing of certain pages

I'd add it to main.tpl wink

  • From: 127.0.0.1
  • Registered: 2001-11-02
  • Posts: 9,167

Re: Preventing search bot indexing of certain pages

I'd use a robots.txt file.

"Programming is like sex: one mistake and you have to support it for the rest of your life."

Rickard's Website