Search options (Page 2 of 8)

Certain characters are escaped, yes - like for example ' becomes \' ... well, it wouldn't work otherwise, so of course

The script has been updated for PunBB v1.2.x - see this thread...

I will, eventually - but for the time being, I haven't even had time to install and test PunBB v1.2 ... too busy in "real life", new girlfriend and stuff - sorry ... I will update it eventually, just don't know at the moment when I'll have time

do you have a specific example? what would you search and replace?

I'm not sure what use that would be? - it operates on entire lines only, you can't replace substrings within lines.

I could support this of course, but it might be risky - what if certain lines in newer versions contain that particular data you were searching for, but it's something you DON'T want replaced? All sorts of random things could happen if we allowed something like this ... also, there's no guarantee that mod authors would be careful enough to ensure that they don't accidentally replace stuff in five different places when they were trying to replace something in just two places.

I think I prefer the search to be as explicit as possible - it's the safest way to avoid carelessness/lazyness from the mod developers ... a search and replace command could really cause havoc.

I didn't know this type of tool even existed - looks very useful ... I'm not sure I have time to get into it now though, I think I should rather concentrate on finishing PunMod first

Well, would there be some way to get your IDs from the query as one comma-separated string, instead of as an array? You need to pass it in comma-separated form to subsequent SQL queries anyways, maybe there is no point in using arrays in the first place?

You should maybe try adding more timers around more blocks of code, not just SQL queries, to discover exactly where the bottleneck is, and then see if there's any way to optimize on it? Of course, for starters, try timing the SQL queries, then compare against the total time taken to generate the entire page, to see how much of the total time is spent by PHP vs how much time is spent by the SQL engine?

Then what is the problem? ... is it simple the number of search queries? which undoubtedly is also a lot higher on forums with that much activity. If so, maybe we could improve things by somehow caching common searches? On a huge forum with a certain topic, there must be lots of searches that get repeated over and over again?

Then how about indexing topics instead of posts? In search_matches, we currently have:

post_id <-> posts.id
word_id <-> search_words.id
subject_match

Suppose we now change it to:

topic_id <-> topics.id
post_id -> posts.id
word_id <-> search_words.id
subject_match

And then store only one record per topic, instead of one per post - the post_id you store for every topic, would be simply the last occurence of the word in each topic ... the way that "search for topics" works, it only displays each topic once, so the information about which other posts may have contained the same word in the same topic, is essentially never used, and as far as I can figure, could be safely discarded?

Of course then, "search for posts" wouldn't be possible - but the output from this mode of search is extremely confusing to browse anyways, as everything is displayed out of context ... I don't know anyone who uses that feature.

Even so, if you wanted to preserve this feature, you could do both - have two tables, one that indexes posts and one that indexes topics ... of course, this would make your search tables possibly take up a considerable amount of extra space. But searches for topics is definitely a lot more common than search for posts - I would guess at least 90% of all searches are probably searches for topics; if those 90% of all searches were working on a table that is probably one fourth of the size (assuming an average 4 posts per topics, which is probably low anyways), that should speed things up considerably.

What do you think?

I'm still here

Major changes have been made to PunMod since the last release - it now works with a transparent backup system, which means it starts out by archiving your whole PunBB installation to a single backup file, from which it restore dynamically; if you don't understand what this means, it basically means you can now add/remove sourcecode changes fully dynamically, e.g. add/remove mods as you please, or undo all mod changes and revert to your original PunBB installation.

However, major work lies ahead - one of the next things I have to do, is of course test and integrate it with PunBB v1.2.

But there is also a major challenge in making database commands which can fully support both install and uninstall.

What this will probably mean, is that commands like ##SQL and ##RUN will have to be dropped. This may sound pretty drastic, but both are virtually impossible to undo, and therefore can't possibly be properly supported. Instead, the ##ADD command will be replace with something more flexible, allowing for table/column/index creation with automatic undo.

Also done is a custom archive format support library (using zlib), currently only used for backup/restore, but which will be used for single-file mod support. Also, subfolders under the punmods folder is supported now, so you can already organize mods and their dependency files more neatly - not that this will be necessary really, once single file "mod archives" are fully supported.

Anyways, that's the status at the moment.

I'll put up a newer version for testing the new improvements at your request.

I meant, adding another indexed column, containing 0 for ids 0-999, 1 for ids 1000-1999, 2 for ids 2000-2999 etc. - this should give a small index for this column, which might help locate the correct group of records faster ... probably the underlying implementation already makes similar optimizations though

Well hey, will you look at that ...

I guess I assumed wrong - even though the search word base is 30% smaller, this apparently makes no real difference in terms of performance at all. In fact, with the stemmer mod enabled, it's slightly slower - I guess the stemming operation itself takes a little time too.

To profile the search engine, I'm executing 20 specific searches in row, and measuring the total time taken to execute each - the 20 searches are executed over again 10 times, and the average total time for those 20 searches calculated.

For added accuracy, I'm executing this test 10 times over, and the shortest total average execution time is the one I use.

Shortest execution time with the stemmer mod enabled: 4.090 seconds

Shortest execution with original search engine: 3.981 seconds

So it's about 2-3% slower.