Indexing is not unicode-safe

Try replacing that line with this:

$result = $db->query('SELECT id, word FROM '.$db->prefix.'search_words WHERE word IN('.implode(',', preg_replace('#^(.*)$#u', '\'\1\'', $unique_words)).')') or error('Unable to fetch search index words', __FILE__, __LINE__, $db->error());

The only difference is the modifier u at the end.

Hmm, I'm sorry, but I have no idea what's going wrong. I have to confess I am far from an expert on localisation.

$unique_words contains a list of all the words in the post to be indexed. The select query then determines which of these words are already indexed so that it can insert only words that are new to the index.

Have you tried it with different browsers? I know it sounds silly, but I know IE handles "exotic" locales differently than e.g. Firefox.

Looks like I nailed it.

The regexes in the split_words functrion were destroying all multibyte text.

Currently I made the following fixes.

1. Went to the .htaccess and set the following:

php_value mbstring.internal_encoding 'UTF-8'
php_value mbstring.func_overload '7'

This way the mbstring extension will overload all string manipulation functions and make the multibyte-safe for us. I assume it is possible to include these calls in locale files of punBB (make a variable called, say, $mbstring_encoding, and set it to 'UTF-8' or whatever). If the mbstring extension is compiled, the respective calls should be made (see http://www.php.net/manual/en/function.m … coding.php).

2. Replaced regexes in split_words with array walks and string functions.
3. Replaced your regexes in index insertion SQL requests with pg_escape_string() calls (it is myltibyte-safe and also a native libpq function that should be faster). Essentially it is a good idea to make a qstr handler in the DBLayer (in a fashion of $db->quote ($words_array))

Seems to work, let me know if you need the code.

Sure it is not. However, instead of declaring that 'punBB is not multibyte-safe' you can make a code fork (not a big one, I mind you).

What I thouhgt is that, essentially, you write in the readme - 'you need it to work multibyte - presto, substitute file X with file Y and you are all set, you need extension Z and extension K for this to work'). People who gonna need this (for example Asian PHP installs) usually have the extension builtin and usually they are familiar with the issue.

PostgreSQL for instance (as well as MySQL of recent versions) both have calls to determine the current client charset and change it. Th only problem I encountered as of yet is with this indexing feature (which is apparently related to the regexps, and it seems that mbstring is not overloading the PCRE extension).

I will send you the code - just make a diff with the one of yours (mine will surely work slowly because I never could grasp regexps).

BTW. a good candidate for a UTF-8 solution is this very forum - just look at these fancy ????? in my post in the russian section :-)

I'm having the same problems with my romanian forum... is this fixed already, or can u Julik sen me a copy of that code... thanx.

I added that htaccess thingy, not sure, but seems fixed, even reindexing works...

Tried some of the changes above.

Looks like the the function strtolower was killing the multibyte characters, so, the regex was not really the problem. I had to replace strtolower  with mb_strtolower in both search_idx.php and search.php. After doing that, the words are inserted into the database properly and searching works too.

PHP5, MySQL 4.1, Apache 2.2, Win32. Will try my changes later on PHP 4.x on Apache 1.3.x / Unix.

So, an mbstring-based fork may be desirable, because hosts do run mbstring.

To summarize what I have seen so far in this forum:

1) 'lang_encoding'    =>'utf-8' in /lang/English/common.php
2) ini_set("default_charset", "utf-8"); in /config.php
3) $db->query("SET NAMES 'UTF8'"); in /include/dblayer/common_db.php
4) utf_general_ci for database
5) word varbinary(20) for 'word' column in 'search_words' table

IMHO (correct me if I am wrong here)

(1) puts charset=utf-8 in mega tags
(2) makes PunBB output charset=utf-8 in HTTP headers.
(3) and (4) allows inserting UTF-8 into the database
(5) gets rid of the error "Unable to insert search index words".

However, there is this code in search_idx.php

    $text = preg_replace($patterns, ' ', ' '.strtolower($text).' ');

which is not Unicode-safe. And at exactly this step, the text gets mangled and after that, the junk is inserted into the search_words table instead of the lowercase words.

In search.php, there is this code

strtolower(trim($_GET['keywords']))

and this passes the junk to the search function.

The strtolower alters the individual bytes (because it assumes that characters are single byte); therefore it is not suitable for altering UTF-8 strings, where some bytes are not altered the same way. When I look into the database with phpMyAdmin, I see question marks inside black losanges instead of Russian words.

The solution posted by Julik is

mbstring.func_overload = 7
mbstring.internal_encoding = UTF-8

Setting mbstring.func_overload to 7 to replace string functions with their multibyte equivalents automatically. I tested it and found it working with PHP5 and .htaccess (because it is INI_PERDIR in PHP5). In PHP4, that parameter is INI_SYSTEM which is not usable in shared hosting environment. In addition to that, if I use that parameter, there is a performance hit. I only need some of the function to be UTF-8-aware, not all of them.

String functions that do not operate on individual characters in strings should be (IMHO) UTF-8 safe, because UTF-8 is backwards-compatible with 8-bit single-byte encoding. Sorting should work too, to some extent (alphabetical case-sensitive Unicode sorting). Functions like changing case, however, are not UTF-8 safe. This is why I replaced strtolower with mb_strtolower function where strtolower occurred.

I just changed varbinary(20) back to varchar(20) and rebuilt the index. The forum still works and I did not get the pesky "Unable to insert search index words" error. So, it looks like varbinary is not necessary. Now I think that varbinary(20) was used to accomodate the characters mangled by strtolower function.

I will do another install (with the latest release instead of the 1.3-dev this time) and then will post my own instructions on how to make PunBB work with UTF-8 (including searching).

And (3) enables inserting and reading to/from database in UTF-8...

I found this page, PHP UTF-8 cheatsheet, which is essentially a quick guide for converting PHP applications to UTF-8.

However, they do not mention the importance of issuing SET NAMES 'utf8' to the database and do not mention the PCRE_UTF8 /u modifier for regular expressions.