Posting in the Magento forums has been disabled pending the implementation of a new and improved forum solution which should better serve the community.

For new questions please post at magento.stackexchange.com, the community-run support site for the Magento community. We will be providing updates on the new forum solution soon. For questions or concerns please email community@magento.com.

Magento Forum

Special Characters Broken After Upgrade
 
kimball1
Jr. Member
 
Total Posts:  11
Joined:  2011-04-02
 

I’ve just upgraded from 1.5.1. to the latest 1.7.0.2.  I’ve sorted out most of the upgrade snafus, but I’m stuck on this one.  My store lists multiple products in multiple languages, and we sell a number of items to international customers.  As a result, we frequently have accented characters in both our catalog (product name, description, etc) as well as in our customer information (customer name, address, etc).  Prior to the upgrade, these characters worked fine, with no special care. After the upgrade, all these characters are converted to a box with a question mark when displayed in the store.

I’ve checked the database contents, and the characters ARE in the database correctly.  They are just not being displayed by the store correctly. As an example:

In the database: Aurículotherapia Interactiva (note the accent on the i)
In the admin product listing: Aur�culoterapia Interactiva

I’ve also checked the database tables, and they are using UTF-8 encoding, and I’ve also adjusted the .htaccess for the site to use UTF-8 as the default charset: AddDefaultCharset UTF-8

What else should I be checking?

Thanks!

 
Magento Community Magento Community
Magento Community
Magento Community
 
kimball1
Jr. Member
 
Total Posts:  11
Joined:  2011-04-02
 

One other bit of information:  I see a LOT of errors in my system.log that look similar to this:

2012-10-18T17:08:03+00:00 ERR (3): Warning: Varien_Simplexml_Element::setNode(): xmlEncodeEntitiesReentrant : char out of range in /var/www/miridiatech/shop/lib/Varien/Simplexml/Element.php on line 481

 
Magento Community Magento Community
Magento Community
Magento Community
 
kalenjordan
Sr. Member
 
Avatar
Total Posts:  218
Joined:  2011-10-31
Pasadena, CA
 

Hey @kimball1, that’s weird.  I just tested it out in a fresh local 1.7.0.0 and it worked fine. 
- Only your product data behaving this way or all content coming from the DB?
- Which attribs specifically are you seeing this with?
- Are the characters html entities or regular characters in the db?

 
Magento Community Magento Community
Magento Community
Magento Community
 
kimball1
Jr. Member
 
Total Posts:  11
Joined:  2011-04-02
 
sparcksoft - 18 October 2012 03:19 PM

Hey @kimball1, that’s weird.  I just tested it out in a fresh local 1.7.0.0 and it worked fine. 
- Only your product data behaving this way or all content coming from the DB?
- Which attribs specifically are you seeing this with?
- Are the characters html entities or regular characters in the db?

Thanks for the questions - here are the answers:
1) I’m seeing this with all kinds of data, not just product data.  For instance, in my shop/var/log/application.log, I’m seeing messages like this:
2012-10-19T15:25:27+00:00 ERR (3): Warning: simplexml_load_string(): iridiatech.com/companyinfo.htm#trademark">Haga clic aquí para Informaci in /var/www/miridiatech/shop/app/code/core/Mage/Core/Model/Config.php on line 522

2) I’m seeing it with everything from product attributes through CMS page contents.

3) In the database, they are regular characters, not html entities.  Many of these have been in the database this way since our very first Magento install (1.3).

 
Magento Community Magento Community
Magento Community
Magento Community
 
kalenjordan
Sr. Member
 
Avatar
Total Posts:  218
Joined:  2011-10-31
Pasadena, CA
 

Interesting. 

4. Only happening with pre-existing content, or will it also happen if you add new content?  For simplicity, maybe test with cms page content (no crazy EAV stuff to worry about for starters).

5. Check the encoding for the database, table, and specific columns.  More info here:
http://www.magentocommerce.com/boards/viewthread/295439/#

I’d be kinda interested to see whether the problem existed if you cloned it down to mageupdate.com - should only take you a couple minutes and would be an interesting test case for me.

 
Magento Community Magento Community
Magento Community
Magento Community
 
kimball1
Jr. Member
 
Total Posts:  11
Joined:  2011-04-02
 
sparcksoft - 19 October 2012 10:51 AM

Interesting. 

4. Only happening with pre-existing content, or will it also happen if you add new content?  For simplicity, maybe test with cms page content (no crazy EAV stuff to worry about for starters).

5. Check the encoding for the database, table, and specific columns.  More info here:
http://www.magentocommerce.com/boards/viewthread/295439/#

I’d be kinda interested to see whether the problem existed if you cloned it down to mageupdate.com - should only take you a couple minutes and would be an interesting test case for me.

As for #4, I have been doing some testing today that is relevant:  I have multiple languages in my store, each set up as their own store using their own locale.  My English, German, French, and Portuguese stores all work fine, but my Spanish store refused to load a all, spewing xml parsing errors into the application log.  I traced these errors down to entries in the core_config table, specifically entries for some of the shipping modes.  In the Spanish versions of these messages, there were lots of é, í, and ó.  When I inspected them in the database, they appeared correctly (using mysql command line as well as in SequelPro).  I also dumped that table into a .csv and opened that in LibreOffice - that worked fine as well.  I even went so far as to truncate the table and re-load for a well-formed .csv correctly encoded using UTF-8.  The store would still puke.  Finally, I replaced all the é with é , í with í, etc.  Now the Spanish store at least loads and is usable.

I’ve found a number of other odd problems that all seem to be linked to UTF-8 encoded characters on various things in the database.  One other one shows up when selecting a state for either a shipping or billing address that is in Germany.  Instead of getting a drop-down menu with the states in it, I get a fill-in box AND a dropdown, but the dropdown has display:none; set in its style tag, and only shows 1 of the German states.  I suspect that whatever is supposed to be populating that drop down pukes on a state with a double-byte character and bails, which then spews out an entry field instead.

So I’m seeing lots of annoying problems all related to string encoding of data after it has been pulled out of the database.

For #5, I have checked the encoding of the database and table, but not the specific columns.  The link you provided with more information seems to just bring me back to this post.... do you have a better link?

I’m interested in tossing it onto mageupdate.  I’ll see if I can follow the instructions there.

Thanks!

 
Magento Community Magento Community
Magento Community
Magento Community
 
kalenjordan
Sr. Member
 
Avatar
Total Posts:  218
Joined:  2011-10-31
Pasadena, CA
 

Sweet.  Will read this in more detail later but I forgot to mention - about the xml parse error - see if you can trace out which file it’s choking on - sometimes the stack traces are more helpful than others though.  I usually resort to xdebug’ing it as a last resort.  Maybe you need to CDATA some xml nodes or something.

I’d like to add stuff like XML integrity checks into the mageupdate tool eventually - and maybe with, you know, half way decent error messages that give you a clue where to look smile

 
Magento Community Magento Community
Magento Community
Magento Community
 
kalenjordan
Sr. Member
 
Avatar
Total Posts:  218
Joined:  2011-10-31
Pasadena, CA
 

Wow, this rabbit whole keeps getting deeper.  Weird that it would throw XML parse errors based on what’s in core_config_data - I guess it merges it all together into an xml object.

Oh sorry for the wrong link, it was a link to a SO post with some specifics on how to grab encodings:
http://stackoverflow.com/a/1049958/1117286

But ya I’m starting to suspect it’s something in the file system / php / apache configuration and not the database.  I’m not sure that I have any more suggestions right now :| Maybe if I had it pulled down somewhere, I could dig a little closer.

Sounds like you’re hot on the trail though!  smile

 
Magento Community Magento Community
Magento Community
Magento Community
Magento Community
Magento Community
Back to top