Topic: punbb getting spidered and indexed
I did the following search:
http://www.google.com/search?hl=en& … gle+Search
to find other forums that google have found. I went through a lot of the results and went digging to find out what pages were being indexed successfully: I want to see how spiderable the current URL format is...or not - and whether the whole mod_rewrite thing is really necessary.
My conclusions were quite clear: user profile pages get indexed in greater numbers and far earlier than any other type of page - including the all important viewtopic pages. Why should this be?
Examples:
http://www.google.com/search?hl=en& … dersson%22
glop.org - 129 pages in index - half of which user profiles - 144 actual posts on forum
this site, punbb, 9000 pages out of 34000 posts
http://www.google.com/search?hl=en& … dersson%22
worrying...2800 posts on this forum, only 73 pages on google and almost all of them user profiles.
if you go through the list of the initial query, you will see the same thing repeated again and again. Any reasons why this should be?