How to fix strange characters in MySQL database

Posted by vinhboy on Friday, May 2nd, 2008 in Internet.

So, as a result of bad programming, my database got FILLED with strange characters like this:

 ’ … – “ †‘ 

These characters are representations of double quotes, single quotes, triple dots, etc… in different encoding formats.
I read that his happens when you copy lines from programs like Word, Wordpad, etc…
It has something to do with using latin-1 as the collation in your database. The many sites that talks about this issue recommends converting your table to use UTF-8.

Anyways, after hours of tedious research, I found a way to fix an existing table that has these weird characters in them.

The code is like this:

update <table_name>
set <field> = replace(<field>,'’','\'');
update <table_name>
set <field>= replace(<field>,'…','...');
update <table_name>
set <field>= replace(<field>,'–','-');
update <table_name>
set <field>= replace(<field>,'“','"');
update <table_name>
set <field>= replace(<field>,'”','"');
update <table_name>
set <field>= replace(<field>,'‘','\'');
update <table_name>
set <field>= replace(<field>,'•','-');
update <table_name>
set <field>= replace(<field>,'‡','c');

replace <table_name> with table name, and <field> with the field you are targeting.

This runs through the database and replaces all the strange characters with normal ones.

9 Comments

  1. Martin Says (October 31st, 2008):

    Thanks for this, handy!

    I squished this together into one query, it works just fine but I’m not sure which is best in terms of efficiency or best practice.

    update set

    =

    replace(replace(replace(
    replace(replace(replace(
    replace(replace(,

    '’', '\''),
    '…', '...'),
    '–', '-'),
    '“', '"'),
    '”', '"'),
    '‘', '\''),
    '•', '-'),
    '‡', 'c');

  2. Woody NaDobhar Says (January 29th, 2009):

    You saved me a lot of time and boredom, my thanks.

  3. Jim Says (February 6th, 2009):

    worked fro me

  4. Paolo Says (March 20th, 2009):

    worked for me also

  5. Yao Says (April 1st, 2009):

    Hey thanks so much, worked for me perfectly!

  6. rvdavid Says (May 26th, 2009):

    I’ve had a similar issue, but found that the culprit was actually utf8 vs latin1 character sets.

    Can you check the following post out and give me some feedback on wether or not this fixes your problem?

    http://blog.rvdavid.net/mysql-funnystrange-characters-replacing-apostrophes-after-import/

    Thanks in advance Vinh.

    rvdavid.

  7. Mobiles Says (September 11th, 2009):

    I was having same problems in my website, thats also worked for me.

    Thanks.

  8. Gerrit Says (January 20th, 2010):

    Sweet, worked flawlessly for me.

  9. n2fisher Says (August 17th, 2010):

    rvdavid lists a possible better solution (not yet tested) but the link is faulty .. he’s removed “blog” from it; the updated link is:

    http://www.rvdavid.net/mysql-funnystrange-characters-replacing-apostrophes-after-import/

Leave a Comment