1 Topic by abclf (edited by abclf 2012-01-05 22:02)

  • abclf
  • Senior Member
  • Offline
  • From: France
  • Registered: 2004-07-30
  • Posts: 148

Topic: Problem with foreign characters π (encoding, utf8)

Hi all !

I recently upgraded my old 1.2.x punbb to last 1.4.1 ;
it looks like it's working, though I'm experiencing an annoying trouble I need to solve :

I can't save foreign characters ; for exemple, π (pi) gives a question mark when printed out after message was saved (preview worked !) ; foreign characters are not saved in database ; and I have been told  phonectic characters didnt work neither (replaced with question mark : ???).

I need help : where to investigate ? what's wrong in my configuration ? what to do ?

(if needed, may be tested there : http://www.languefrancaise.net/forum/vi … .php?id=9)

As for sql, all punbb informations are stocked in a single database where all tables (excepted old tables not in use anymore) are utf8_general_ci ; nothing about encoding in my config.

Looking at the files in 1.4.1 I was surprised to note that some appear to be utf8 (register.php) and some windows1252 encoded.

Thank you !

images : http://imgur.com/a/inY0c#0

abclf's Website

  • From: Ukraine
  • Registered: 2008-10-31
  • Posts: 762

Re: Problem with foreign characters π (encoding, utf8)

Can you post test message with foreign characters here?

3 Reply by abclf

  • abclf
  • Senior Member
  • Offline
  • From: France
  • Registered: 2004-07-30
  • Posts: 148

Re: Problem with foreign characters π (encoding, utf8)

Yes wink

example : d likes ৪-১ এ জিতল ম্যারাডোনার শিষ্য মেসি হিগুয়েনরা! (পাতা ১) - ফুটবল - খেলাধূলা - প্রজন্ম ফোরাম on foru

Hum, when upgrading my old punbb I have had to migrate the data sql to a new database (export big sql file, upload it, use  http://www.ozerov.de/bigdump/ to run the import in new database, upload new punbb files, run update, change config.php).

The new database is utf-8, tables are utf-8, I don't see anything suspect.
I can write foreign characters in text area and show them nicely in preview (they are processed?) ; when I save (different process?), they are not encoded, just transformed in ????
I'm not sure this trouble is related to punbb ; but, if not, what could be the cause, and how to test (php ? apache ?) ?

abclf's Website

  • From: Ukraine
  • Registered: 2008-10-31
  • Posts: 762

Re: Problem with foreign characters π (encoding, utf8)

I think problem in DB.

5 Reply by abclf

  • abclf
  • Senior Member
  • Offline
  • From: France
  • Registered: 2004-07-30
  • Posts: 148

Re: Problem with foreign characters π (encoding, utf8)

Do you have any idea about how to test/debug ? or how to get db informations wich could lead us to solve the problem ?

Here it is a structure export for post table :

-- phpMyAdmin SQL Dump
-- version 2.6.4-pl3
-- http://www.phpmyadmin.net
-- 
-- Serveur: xxx.db.1and1.com
-- Généré le : Vendredi 06 Janvier 2012 à 12:07
-- Version du serveur: 5.0.91
-- Version de PHP: 5.3.3-7+squeeze3
-- 
-- Base de données: `xxx`
-- 

-- --------------------------------------------------------

-- 
-- Structure de la table `punbb_posts`
-- 

CREATE TABLE `punbb_posts` (
  `id` int(10) unsigned NOT NULL auto_increment,
  `poster` varchar(200) character set latin1 NOT NULL default '',
  `poster_id` int(10) unsigned NOT NULL default '1',
  `poster_ip` varchar(39) character set latin1 default NULL,
  `poster_email` varchar(80) character set latin1 default NULL,
  `message` text character set latin1,
  `hide_smilies` tinyint(1) NOT NULL default '0',
  `posted` int(10) unsigned NOT NULL default '0',
  `edited` int(10) unsigned default NULL,
  `edited_by` varchar(200) character set latin1 default NULL,
  `topic_id` int(10) unsigned NOT NULL default '0',
  PRIMARY KEY  (`id`),
  KEY `punbb_posts_topic_id_idx` (`topic_id`),
  KEY `punbb_posts_multi_idx` (`poster_id`,`topic_id`),
  KEY `punbb_posts_posted_idx` (`posted`)
) ENGINE=MyISAM AUTO_INCREMENT=89636 DEFAULT CHARSET=utf8 AUTO_INCREMENT=89636 ;
-- 
-- Structure de la table `punbb_config`
-- 

CREATE TABLE `punbb_config` (
  `conf_name` varchar(255) character set latin1 NOT NULL default '',
  `conf_value` text character set latin1,
  PRIMARY KEY  (`conf_name`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;

-- --------------------------------------------------------

abclf's Website

  • From: Ukraine
  • Registered: 2008-10-31
  • Posts: 762

Re: Problem with foreign characters π (encoding, utf8)

Maybe `message` text character set latin1.
Why latin1?

7 Reply by abclf

  • abclf
  • Senior Member
  • Offline
  • From: France
  • Registered: 2004-07-30
  • Posts: 148

Re: Problem with foreign characters π (encoding, utf8)

lol ; I dont know why, it's not my own choice. And yes, of course, it is suspect.
I assume this is the unaltered setting coming from the old punbb database ; maybe update.php has not changed that (although it was very loooong to process).

What's expected for this table ? Is there somewhere I can see the database structure created by a new punbb install ?

Thank you for your help wink

abclf's Website

  • From: Ukraine
  • Registered: 2008-10-31
  • Posts: 762

Re: Problem with foreign characters π (encoding, utf8)

Default without any charset, but with charset for table.

  • Registered: 2011-12-26
  • Posts: 38

Re: Problem with foreign characters π (encoding, utf8)

The Latin alphabet is the once that many languages have, including French, English and Dutch. When using a Cyrillic character set, Latin letters will produce question marks. I'm afraid it can only be resolved by using UTF-8 or UTF-16. I don't know if PunBB supports UTF-8 and UTF-16 though.

  • From: Ukraine
  • Registered: 2008-10-31
  • Posts: 762

Re: Problem with foreign characters π (encoding, utf8)

PunBB 1.4 support ONLY UTF-8.

11 Reply by abclf

  • abclf
  • Senior Member
  • Offline
  • From: France
  • Registered: 2004-07-30
  • Posts: 148

Re: Problem with foreign characters π (encoding, utf8)

Hello,

I have manually altered my database : everywhere a field's charset was set as latin 1 swedish, I deleted the setting (edit fied > choose [blank] in the list).
After saving, the field's charset is automatically altered in utf8 general ci.
This should allow saving exotic characters in the future.

It looks like it is working as expected now. Thanks for the help.

If this doesn't exist, I would ask for a wiki page containing the sql structure of pristine punbb install.

abclf's Website