Topic: Preventing search bot indexing of certain pages

There are some pages in a bulletin board that you don't want search engines to index. Some examples are: post.php, login.php, search.php etc etc. You basically want search engines to only index the pages that have real content, e.g. index.php, viewforum.php, viewtopic.php and profile.php. A solution to the problem is to add a meta element to the pages that we don't want indexed? Something like this:

<meta name="ROBOTS" content="NOINDEX" />

How does that sound?

"Programming is like sex: one mistake and you have to support it for the rest of your life."

Re: Preventing search bot indexing of certain pages

Sounds like a good idea.

Are there any drawbacks using this method?

Re: Preventing search bot indexing of certain pages

That's why I asked. I have no idea :)

Edit: Here's some reading.

"Programming is like sex: one mistake and you have to support it for the rest of your life."

4

Re: Preventing search bot indexing of certain pages

Couldn't you use a robots.txt file?

Re: Preventing search bot indexing of certain pages

A robots.txt is an alternative. I just thought the meta tag thing was a bit more elegant. Maybe not.

"Programming is like sex: one mistake and you have to support it for the rest of your life."

Re: Preventing search bot indexing of certain pages

hmm... ill try it when i get my site on google. ive never had a site in a search engine before, so i have almost no clue how this stuff works sad

Indocron
$theQuestion = (2*b) || !(2*b);

Re: Preventing search bot indexing of certain pages

Gary, visit robotstxt.org for more info

Re: Preventing search bot indexing of certain pages

i would suggest to EXCLUDE the profile.php also so as to not allow bots to collect emails also.  just a suggestion.

~James
FluxBB - Less is more

9 (edited by snapsolutions 2004-09-22 00:42)

Re: Preventing search bot indexing of certain pages

robots can't fetch your email address from the profile menu

Re: Preventing search bot indexing of certain pages

Unless you explicitly choose to have your e-mail address publicly visible.

"Programming is like sex: one mistake and you have to support it for the rest of your life."

Re: Preventing search bot indexing of certain pages

any i doubt a meta-tag stops email harvestors

Re: Preventing search bot indexing of certain pages

here's a good explanation how to write robots.txt files
http://www.robotstxt.org/wc/norobots.html

Re: Preventing search bot indexing of certain pages

There's a downside to using a robots.txt, it must be accessible via the URL "/robots.txt". That means that it will have be placed in the domain or subdomain root directory even if the forums are accessed via e.g. "/forums/". This in turn means that robots.txt will have to be edited depending on in which directory the forums are located.

"Programming is like sex: one mistake and you have to support it for the rest of your life."

Re: Preventing search bot indexing of certain pages

yep ... unfortunately, but at least many robots search for it, so I guess they use it aswell wink

Re: Preventing search bot indexing of certain pages

1 question.. why would you want to stop robots scanning your pages? surely the more pages the better, even if they are not very useful

Re: Preventing search bot indexing of certain pages

I hate searching for something and finding the response in e.g. a post form instead of the actual topic.

"Programming is like sex: one mistake and you have to support it for the rest of your life."

17

Re: Preventing search bot indexing of certain pages

For the bots that DO follow the rules you should use:

<meta name="ROBOTS" content="NOINDEX,FOLLOW" />

which will tell the bots not to index the current page but to follow the links on the page to other pages.

For those bots that don't play nice I ban them using htaccess.

Re: why? I don't need my bandwidth being eaten up by a rogue bot, email harvester, website whacker, or some wannabe hacker chewing through the pages on my forum.

18

Re: Preventing search bot indexing of certain pages

Some one found my site by search for a Joke in google,  that joke found in my forum in "General discussions" forum, i hate that 

I think it is good to execlude "General discussions" in that way, maybe add option to forum options.

If your people come crazy, you will not need to your mind any more.

Re: Preventing search bot indexing of certain pages

lorax: Good idea.

zaher: That, on the other hand, will require a forum specific option and I'm not sure it's worth it.

"Programming is like sex: one mistake and you have to support it for the rest of your life."

Re: Preventing search bot indexing of certain pages

zaher thats a bit bad because your genral discussions might just happen to have the joke they are looking for wink

Re: Preventing search bot indexing of certain pages

If we remove the meta tag of keyword and description from our pages, still the search engine will spot it or excluively we have to define like <meta name="ROBOTS" content="NOINDEX" />

God wisely designed the human body so that we can neither pat our own backs nor kick ourselves too easily

Re: Preventing search bot indexing of certain pages

This has now been implemented in 1.2. The meta tag

<meta name="ROBOTS" content="NOINDEX, FOLLOW" />

can be found on alla pages except: index.php, viewforum.php, viewtopic.php, profile.php (viewing a profile as a guest) and userlist.php. Apparently, not all robots support the meta tag, but the important ones do (i.e. Googlebot :D).

"Programming is like sex: one mistake and you have to support it for the rest of your life."

23 (edited by nrwilk 2006-04-15 17:06)

Re: Preventing search bot indexing of certain pages

May I ask how I can block search bots from scanning ANY forum pages?

Should I just add

<meta name="ROBOTS" content="NOINDEX, NOFOLLOW" />

to the top of all pages associated with PunnBB?

Thanks!

Re: Preventing search bot indexing of certain pages

I'd add it to main.tpl wink

Re: Preventing search bot indexing of certain pages

I'd use a robots.txt file.

"Programming is like sex: one mistake and you have to support it for the rest of your life."