1 Topic by Yann

  • Yann
  • Senior Member
  • Offline
  • Registered: 2004-04-16
  • Posts: 125

Topic: Improve search?

Hello, I wanted to ask here if there were plans to improve the search, or if someone actually already did. The search is poor, as in its performance as in its results; when I look for nvidia on our forum, I get over 277 pages of unweighted results, or more complex queries i dont get any answer at all, because it must return too much results, hitting some of my mysql limits, certainly.

Has someone already tried to bundle punbb with another search engine, like lucene, or any other one, that would actually be faster and more effective?

Yann's Website

  • From: Leuven, Belgium
  • Registered: 2005-06-14
  • Posts: 2,653

Re: Improve search?

1.3 will use fulltext searching, that will gravely optimise the search

  • Smartys
  • Former PunBB Developer
  • Offline
  • Registered: 2004-04-30
  • Posts: 7,844

Re: Improve search?

Yann: If you're getting a lot of poor results, I'd suggest a couple things
A. http://punbb.org/forums/viewtopic.php?id=3884
Add more stopwords to the list to try and eliminate noise.
B. Start the search reindexing in the admin panel

Personally, I find PunBB always returns great results, but as Bekko said the efficency of the searching (and hopefully the relevance) for users of MySQL will be increased greatly in 1.3

4 Reply by Yann

  • Yann
  • Senior Member
  • Offline
  • Registered: 2004-04-16
  • Posts: 125

Re: Improve search?

Smartys > The search indexer needs to be completely rewritten too imho... Reindexing would take up to one week on our servers with their current load (took me 2 days last year wih 1/3 of the posts and less loaded servers) - and I don't feel like keeping a browser window open all the time.

Concerning 1.3.... The last time I heard punbb 1.3 would come out, it was in september 2005, stating that it would probably come out before the end of the year... And I think if 1.3 goes the same way as 1.2, with beta 1->4, it will take at least one other year from now on to get it production-ready tongue

There was a fullsearch mod for punbb 1.2, is it working well? Is there a backport of the fulltext search from 1.3 to 1.2?

Yann's Website

  • Smartys
  • Former PunBB Developer
  • Offline
  • Registered: 2004-04-30
  • Posts: 7,844

Re: Improve search?

There is no backport, and I don't know how the mod is wink

And as for the reindexing, how you would you suggest it be done faster (I know of at least one change I want to make to it)? tongue

6 Reply by Yann

  • Yann
  • Senior Member
  • Offline
  • Registered: 2004-04-16
  • Posts: 125

Re: Improve search?

Smartys > for the reindexing I would suggest a tool in perl/php-cli, so that it can be run from the command line in a screen... With an option for the speed of the queries.
Concerning the search, we are still having _huge_ problems with it, because of its performance and the pertinence of the results. We will see if we find a solution...

Yann's Website

  • Smartys
  • Former PunBB Developer
  • Offline
  • Registered: 2004-04-30
  • Posts: 7,844

Re: Improve search?

Yann wrote:

Smartys > for the reindexing I would suggest a tool in perl/php-cli, so that it can be run from the command line in a screen... With an option for the speed of the queries.

The queries are already run as fast as possible and you can control how many topics are indexed in one pageview. And running from a shell is not something everyone can do. I don't really see how that would improve indexing.

8 Reply by Yann (edited by Yann 2007-06-10 10:13)

  • Yann
  • Senior Member
  • Offline
  • Registered: 2004-04-16
  • Posts: 125

Re: Improve search?

Smartys > Exactly, the queries are run as fast as possible, which may not be what you want... Reindexing is extremely hard on the sql server, personnally i would prefer having the reindex process take more time with less load on the sql server. Concerning the shell version, I agree this is not something everyone can do, but:
1) that would be interesting for forums with several 100k messages, and at that size you usually have a shell account
2) Believe me, when the reindexation process takes several DAYS, you don't want to let your firefox open. Moreover, imagine you have a network failure within these three days, the reindexation process just crashes, and you will have to restart it manually where it had stopped. Same if your firefox just crashes for some reason...

So when I speak about improving the indexation process, dont misunderstand me: I don't speak of speeding up the process, I ask for a somewhat more reliable way to do it.

You should try to reindex a big forum with the current system tongue

Yann's Website

  • Smartys
  • Former PunBB Developer
  • Offline
  • Registered: 2004-04-30
  • Posts: 7,844

Re: Improve search?

Exactly, the queries are run as fast as possible, which may not be what you want... Reindexing is extremely hard on the sql server, personnally i would prefer having the reindex process take more time with less load on the sql server.

But weren't you saying that reindexing was taking too long? wink

Believe me, when the reindexation process takes several DAYS, you don't want to let your firefox open. Moreover, imagine you have a network failure within these three days, the reindexation process just crashes, and you will have to restart it manually where it had stopped. Same if your firefox just crashes for some reason...

Same if your server restarts for some reason tongue

You should try to reindex a big forum with the current system

How big? I reindexed a 150k post forum in under a day tongue