I had the same problem.
sample log
91.224.160.11 - - [16/Oct/2011:00:40:46 +0400] "GET /forum/new-reply90.html%09-%09200%094%09-%09%CA%EE%EC%EF%FC%FE%F2%E5%F0%FB%09-%09-%09-%09-%09-%09- HTTP/1.0" 404 480 "http://www.datarc.ru/forum/new-reply90.html%09-%09200%094%09-%09%CA%EE%EC%EF%FC%FE%F2%E5%F0%FB%09-%09-%09-%09-%09-%09-" "Mozilla/6.0 (compatible; MSIE 7.0a1; Windows NT 5.2; SV1)"
temporarily fixed this way, but do not think it is good way to patch functions.php directly:
function forum_htmlencode($str)
{
$return = ($hook = get_hook('fn_forum_htmlencode_start')) ? eval($hook) : null;
if ($return != null)
return $return;
//// return htmlspecialchars($str, ENT_QUOTES, 'UTF-8');
return htmlspecialchars(mb_convert_encoding($str, 'UTF-8', mb_detect_encoding($str)), ENT_QUOTES, 'UTF-8');
}