1

Topic: Bad character encoding after a migration

Ok, I had to migrate a french speaking punBB forum to a new server.

Everything is working fine, except for the character encoding. Certain characters (such as à, for instance), do not appear correctly, which completely mangles the text. This is true for the text in the menus (in french), and also in the texts I write.

This was not a problem in my previous server. Actually, all the posts made in that server are still encoded perfectly well. It's only the posts written in punBB on the new server that are mangled (and the menu's text).

Does anyone have a clue of what's going on, and how can I adress this? I don't understand why punBB was producing perfectly fine text, and now sputters so much garbage.

Re: Bad character encoding after a migration

Link?

3

Re: Bad character encoding after a migration

Smartys wrote:

Link?

Here it goes:

http://lunerouge.ochipepe.org/forum/index.php

Initially everything looks fine (lots of accented characters correctly encoded), but you can see for instance that instead of "Déconnexion", there is "D,connexion", and other similar mistakes. The worse is when you type new posts....

Re: Bad character encoding after a migration

It looks to me like the language files are in ISO-8859-1, whereas the database is encoded in UTF-8

5

Re: Bad character encoding after a migration

Smartys wrote:

It looks to me like the language files are in ISO-8859-1, whereas the database is encoded in UTF-8

That's possible... But in that case, why are the new posts also faulty? they should be encoded correctly, no?

6

Re: Bad character encoding after a migration

Maybe unimportant, but actually the "example" plug-in writes perflectly the text I give it to him, no matter what accents I use...

Re: Bad character encoding after a migration

Is the old site still up?

8

Re: Bad character encoding after a migration

yup: http://lune.rouge.free.fr/forum/index.php

Re: Bad character encoding after a migration

What's up with the huge text size? >=/

Anyway, the message preview works because the text hasn't been in the database yet wink

10

Re: Bad character encoding after a migration

elbekko wrote:

[
What's up with the huge text size? >=/

We use big screens wink

Anyway, the message preview works because the text hasn't been in the database yet wink

I see... But once again, if the problem is in the database, why would the older posts not be affected?

This is pretty frustrating

Re: Bad character encoding after a migration

AlexS wrote:
elbekko wrote:

What's up with the huge text size? >=/

We use big screens wink

So do I, and it looks horrible tongue

I see... But once again, if the problem is in the database, why would the older posts not be affected?

Check what the default encoding of your database is.

12

Re: Bad character encoding after a migration

elbekko wrote:

So do I, and it looks horrible tongue

Actually you made me have a closer look at it, and when I see the forum, it looks exactly like this one (forums.punbb.org), except I'm using the Lithium theme. The font is also the same size, so I don't get what's the problem? That's strange, I'll ask other users for their feed back. But this is besides the point.

Check what the default encoding of your database is.

Do you know which command should I issue?

Re: Bad character encoding after a migration

AlexS wrote:
elbekko wrote:

So do I, and it looks horrible tongue

Actually you made me have a closer look at it, and when I see the forum, it looks exactly like this one (forums.punbb.org), except I'm using the Lithium theme. The font is also the same size, so I don't get what's the problem? That's strange, I'll ask other users for their feed back. But this is besides the point.

I'm on a 21.3" screen with 1600x1200 resolution, and the text size is much, much bigger than on these forums wink

Do you know which command should I issue?

You should be able to see this in phpMyAdmin.

14

Re: Bad character encoding after a migration

You should be able to see this in phpMyAdmin.

My server does not have phpMyAdmin installed, so I'm forced to do direct mysql commands, which complicates my life. I'm trying to find out how to get that info

15

Re: Bad character encoding after a migration

Ok, I looked at an SQL back up of my previous database, and I find this:

-- CREATE DATABASE `lune_rouge` DEFAULT CHARACTER SET latin1 COLLATE latin1_general_ci;
-- USE lune_rouge;

-- --------------------------------------------------------

--
-- Structure de la table `forum1_bans`
--

CREATE TABLE `forum1_bans` (
  `id` int(10) unsigned NOT NULL auto_increment,
  `username` varchar(200) collate latin1_general_ci default NULL,
  `ip` varchar(255) collate latin1_general_ci default NULL,
  `email` varchar(50) collate latin1_general_ci default NULL,
  `message` varchar(255) collate latin1_general_ci default NULL,
...

So the previous encoding was latin1

When I do a dump of my current mysql db I get:

-- MySQL dump 10.10
--
-- Host: localhost    Database: ochipepe_LRforum
-- ------------------------------------------------------
-- Server version       5.0.22-Debian_3-log

/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;
/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;
/*!40101 SET NAMES utf8 */;

So I think now it's utf8...

If that's so, is there an easy way out, or should I dump the database and start the recovery from scratch?

Also, I would like the database to be encoded in utf-8, to avoid such problems in the future. Is there a way of doing that?

16 (edited by elbekko 2006-09-02 16:02)

Re: Bad character encoding after a migration

ALTER DATABASE `game` DEFAULT CHARACTER SET utf8 COLLATE utf8_unicode_ci

Maybe? That's what phpMyAdmin gave me... (I performed this operation on an old database tongue)

17

Re: Bad character encoding after a migration

Did phpMyAdmin do the conversion in a smart way, or did it mangle completely the text? Did you check that?

Re: Bad character encoding after a migration

AlexS wrote:

Did phpMyAdmin do the conversion in a smart way, or did it mangle completely the text? Did you check that?

Nope, it was an unused database. I don't feel like installing yet another installation of PunBB on my localhost now...

19

Re: Bad character encoding after a migration

Ok, so I decided to start from scratch, and I dumped the new DB

But this time, instead of just creating the table ochipepe_LRforum, I used the following command:

CREATE DATABASE `ochipepe_LRforum` DEFAULT CHARACTER SET latin1 COLLATE latin1_general_ci;

and then injected the backed up DB

When I check the dump, I see the following:

-- MySQL dump 10.10
--
-- Host: localhost    Database: ochipepe_LRforum
-- ------------------------------------------------------
-- Server version       5.0.22-Debian_3-log

/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;
/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;
/*!40101 SET NAMES utf8 */;
/*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */;
/*!40103 SET TIME_ZONE='+00:00' */;
/*!40014 SET @OLD_UNIQUE_CHECKS=@@UNIQUE_CHECKS, UNIQUE_CHECKS=0 */;
/*!40014 SET @OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0 */;
/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='NO_AUTO_VALUE_ON_ZERO' */;
/*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */;

--
-- Table structure for table `forum1_bans`
--

DROP TABLE IF EXISTS `forum1_bans`;
CREATE TABLE `forum1_bans` (
  `id` int(10) unsigned NOT NULL auto_increment,
  `username` varchar(200) collate latin1_general_ci default NULL,
  `ip` varchar(255) collate latin1_general_ci default NULL,
  `email` varchar(50) collate latin1_general_ci default NULL,
  `message` varchar(255) collate latin1_general_ci default NULL,
  `expire` int(10) unsigned default NULL,
  PRIMARY KEY  (`id`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 COLLATE=latin1_general_ci;
...

First, somehow it seems SET NAMES is in UTF8, despite my command.
Second, the tables are created as latin1_general_ci, so that should be fine

However, I still have the same problem with my forum: the menus are mangled, the new posts also, but not the old ones.

Now I'm totally lost. Any ideas?