I had the same problem.

sample log

91.224.160.11 - - [16/Oct/2011:00:40:46 +0400] "GET /forum/new-reply90.html%09-%09200%094%09-%09%CA%EE%EC%EF%FC%FE%F2%E5%F0%FB%09-%09-%09-%09-%09-%09- HTTP/1.0" 404 480 "http://www.datarc.ru/forum/new-reply90.html%09-%09200%094%09-%09%CA%EE%EC%EF%FC%FE%F2%E5%F0%FB%09-%09-%09-%09-%09-%09-" "Mozilla/6.0 (compatible; MSIE 7.0a1; Windows NT 5.2; SV1)"

temporarily fixed this way, but do not think it is good way to patch functions.php directly:

function forum_htmlencode($str)
{
    $return = ($hook = get_hook('fn_forum_htmlencode_start')) ? eval($hook) : null;
    if ($return != null)
        return $return;

////    return htmlspecialchars($str, ENT_QUOTES, 'UTF-8');
    return htmlspecialchars(mb_convert_encoding($str, 'UTF-8', mb_detect_encoding($str)), ENT_QUOTES, 'UTF-8');
}