Magento Forum

Accepting “international keyboard” special character entry in Magento SOLVED (see response #4)
 
Grayson
Member
 
Avatar
Total Posts:  58
Joined:  2007-11-20
Netherlands
 

From this thread on Unicode support (response #19):

Grayson - 10 January 2008 12:40 PM

I don’t know how much work is involved in changing the tiny_mce distribution with Magento to accept the “international keyboard” input method (e.g., double quotation mark + e = ë), but I’ll look further tomorrow.

After a cursory glance, it looks like the charset has been changed to utf-8 in the tinyMCE distribution in Magento. However, in Western Europe the text typed into the textarea (say, a product description entry area or text area on a CMS page) by the user will be latin-1 (ISO-8859-1) replete with ë‘s and à‘s entered with a few keyboard strokes and not as HTML entities.

So as a first stab I’d say that what needs to happen is that the input in the textarea must be captured and converted from latin-1 to html entities before tinyMCE sees it, and vice versa when tinyMCE sends text from the database to the screen--grab that text and convert any html entities to latin-1 characters before display.

Whatever the source of the problem and the corrrect fix may be, this is very important for European users. It’ll be veeeerrrrry tiresome to have to type in & euml; everytime you want ë--not to mention that site visitors typing in tags, reviews, questions, and so on should be able to type “normally” and see the proper output.

Can someone from the Magento team shed some wisdom on this issue? I’m not familiar with the tinyMCE code and have only barely dipped a toe into the inner workings of Magento--the team can undoubtedly fix this muuuuuuuuuch faster than I could smile.

 
Magento Community Magento Community
Magento Community
Magento Community
 
Grayson
Member
 
Avatar
Total Posts:  58
Joined:  2007-11-20
Netherlands
 

I tried to add a tag to a product as an innocent potential customer browsing my site: ”gááf”. This is what I, the hapless visitor, saw next:

Noticeiconv_strlen() [function.iconv-strlen]Detected an illegal character in input string in /home/butt1348/butterwing.nl/www/app/code/core/Mage/Tag/Model/Mysql4/Tag.php on line 41
[0] in iconv_strlen
("gááf""UTF-8"in /home/butt1348/butterwing.nl/www/app/code/core/Mage/Tag/Model/Mysql4/Tag.php on line 41
[1] in Mage_Tag_Model_Mysql4_Tag
->loadByName(Mage_Tag_Model_Tag"gááf"in /home/butt1348/butterwing.nl/www/app/code/core/Mage/Tag/Model/Tag.php on line 41
[2] in Mage_Tag_Model_Tag
->loadByName("gááf"in /home/butt1348/butterwing.nl/www/app/code/core/Mage/Tag/controllers/IndexController.php on line 60
[3] in Mage_Tag_IndexController
->saveAction() in /home/butt1348/butterwing.nl/www/app/code/core/Mage/Core/Controller/Varien/Action.php on line 332
[4] in Mage_Core_Controller_Varien_Action
->dispatch("save"in /home/butt1348/butterwing.nl/www/app/code/core/Mage/Core/Controller/Varien/Router/Standard.php on line 149
[5] in Mage_Core_Controller_Varien_Router_Standard
->match(Mage_Core_Controller_Request_Httpin /home/butt1348/butterwing.nl/www/app/code/core/Mage/Core/Controller/Varien/Front.php on line 166
[6] in Mage_Core_Controller_Varien_Front
->dispatch() in /home/butt1348/butterwing.nl/www/app/Mage.php on line 411
[7] in Mage
::run("base"in /home/butt1348/butterwing.nl/www/index.php on line 29

Notice
iconv_strlen() [function.iconv-strlen]Detected an illegal character in input string in /home/butt1348/butterwing.nl/www/app/code/core/Mage/Tag/Model/Mysql4/Tag.php on line 65
[0] in iconv_strlen
("gááf""UTF-8"in /home/butt1348/butterwing.nl/www/app/code/core/Mage/Tag/Model/Mysql4/Tag.php on line 65
[1] in Mage_Tag_Model_Mysql4_Tag
->_beforeSave(Mage_Tag_Model_Tagin /home/butt1348/butterwing.nl/www/app/code/core/Mage/Core/Model/Mysql4/Abstract.php on line 335
[2] in Mage_Core_Model_Mysql4_Abstract
->save(Mage_Tag_Model_Tagin /home/butt1348/butterwing.nl/www/app/code/core/Mage/Core/Model/Abstract.php on line 209
[3] in Mage_Core_Model_Abstract
->save() in /home/butt1348/butterwing.nl/www/app/code/core/Mage/Tag/controllers/IndexController.php on line 65
[4] in Mage_Tag_IndexController
->saveAction() in /home/butt1348/butterwing.nl/www/app/code/core/Mage/Core/Controller/Varien/Action.php on line 332
[5] in Mage_Core_Controller_Varien_Action
->dispatch("save"in /home/butt1348/butterwing.nl/www/app/code/core/Mage/Core/Controller/Varien/Router/Standard.php on line 149
[6] in Mage_Core_Controller_Varien_Router_Standard
->match(Mage_Core_Controller_Request_Httpin /home/butt1348/butterwing.nl/www/app/code/core/Mage/Core/Controller/Varien/Front.php on line 166
[7] in Mage_Core_Controller_Varien_Front
->dispatch() in /home/butt1348/butterwing.nl/www/app/Mage.php on line 411
[8] in Mage
::run("base"in /home/butt1348/butterwing.nl/www/index.php on line 29
So Magento is assuming text input from a visitor is in utf-8, and standard European accented characters such as á don’t fly.

This is not a tinyMCE thing; the input is submitted via a standard HTML <form> with some js validation. I don’t see how it can be assumed that a random visitor will be typing in input in utf-8; it seems to me that all these points of user input are going to have to be assumed to be something else (in Western European stores catering to Western European visitors, probably latin-1, but obviously this will need to be tailored to the shop and its audience), intercepted before further Magento processing, and converted to utf-8—or to ASCII with html entities for the “illegal” characters such as á; these are rendered correctly in the storefront by the browser (though they show up as the html entity descpription in the database and the admin).

Since Magento is a US project catering to an English-speaking audience, this issue hasn’t been readily visible (as English is contained within ASCII which is a subset of utf-8). But for Magento to work for stores with a non-English-speaking presence, it must be addressed.

 
Magento Community Magento Community
Magento Community
Magento Community
 
SeL_
Magento Team
 
Avatar
Total Posts:  1140
Joined:  2007-10-10
Paris, France
 

Hi Grayson,

I’ve tried your example to add a tag ‘gááf’ with special characters (with ñ and ë) and it worked out fine. I could create the tag. In the admin, i was able to see it and validate it. And then it was on the frontend. The tag display is correct.
I’ve done it under :
- firefox 2 on ubuntu
- opera on ubuntu
- firefox 2 on XP
- IE 6.0 on XP
- IE 7.0 on XP
On all of them, it was ok.

In my html code, i have:
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
between <head> tags.
This is included in app/design/adminhtml/default/default/template/page.phtml

I can also see that encoding of my document is UTF-8 (you can check it by right clicking on your page in your browser and select ‘show page information’ or something like that: see screen capture).
Sometimes Apache overwrites encoding if AddDefaultCharset has been set. I didn’t set this option in my configuration. Did you ?

Can you check that your configuration is ok ?

BTW, my magento version is the latest: 0.7.148

Image Attachments
Capture-Informations sur la page.png
 
Magento Community Magento Community
Magento Community
Magento Community
 
Grayson
Member
 
Avatar
Total Posts:  58
Joined:  2007-11-20
Netherlands
 

SeL, thanks for your response. My page character encoding is set to utf-8 in my interface/theme html header, and this is what my shop pages show in the HTML header “content” section.

The server is a different story. Using phpinfo I see the following:

Apache Environment
Variable    Value
HTTP_ACCEPT_CHARSET     ISO
-8859-1,utf-8;q=0.7,*;q=0.7

HTTP Request Headers
Accept
-Charset     ISO-8859-1,utf-8;q=0.7,*;q=0.7

HTTP Response Headers
XContent
-Type     text/htmlcharset=ISO-8859-1

I think my problem lies in the HTTP response header content type (last line above). I didn’t set the content type in the file that contains the phpinfo() call, and the server is assuming ISO-8859-1. So it does indeed look like the character set may be being overriden by the server settings. I’ll contact my hoster and have them check and/or fix this.

Merci beaucoup smile for the Apache tip. You have no idea how glad I am this is a server problem and not a Magento problem!!!!!!

 
Magento Community Magento Community
Magento Community
Magento Community
 
Grayson
Member
 
Avatar
Total Posts:  58
Joined:  2007-11-20
Netherlands
 

It’s actually easier than having to contact my hoster: I just added the line

AddDefaultCharset UTF-8

to the .htaccess file in the magento root directory. Solved the entire problem. See this article for a lot of good information about this.

I suggest that this line be added to the Magento release to avoid this problem coming up for others (since the entire app is in utf-8 anyway). I’ll add that suggestion to the bug report I’d posted a comment in earlier.

SeL, thanks again for your wonderful tip!

 
Magento Community Magento Community
Magento Community
Magento Community
 
SeL_
Magento Team
 
Avatar
Total Posts:  1140
Joined:  2007-10-10
Paris, France
 

Just a comment.
As it’s written in apache documentation, if you add this AddDefaultCharset option in your .htaccess file, it will override encoding of all outgoing content.
It’s better to let it done by our application (Magento) and make sure that the web server doesn’t override it.

But if you are sure that all your content will be UTF-8 encoded, you won’t have problems with this solution.

Happy to be helpful wink

 
Magento Community Magento Community
Magento Community
Magento Community
 
Grayson
Member
 
Avatar
Total Posts:  58
Joined:  2007-11-20
Netherlands
 

I’ve assumed all Magento-generated content will be in utf-8, and modified the .htaccess file in the magento/ root. Doesn’t the .htaccess file limit the AddDefaultCharset to the directory tree it’s located in?

The problem is that if people host their Magento shop on third-party servers that have AddDefaultCharset on and set to something other than utf-8, they’ll encounter the above problem no matter what Magento tries to do to fix it. It’s great, too, if the individual user can fix this problem locally instead of needing to go through his or her hoster (server owner) to fix it. Not to mention which lots of people who want to use Magento in the future won’t even know what all this means . . . so I’d definitely like to see the modification to the .htaccess file incorporated into future Magento releases.

 
Magento Community Magento Community
Magento Community
Magento Community
 
Moshe
Magento Team
 
Avatar
Total Posts:  1770
Joined:  2007-08-07
Los Angeles
 

I’ve added it commented out with note that it should be tried if there are problems with UTF encoding smile

 
Magento Community Magento Community
Magento Community
Magento Community
 
SeL_
Magento Team
 
Avatar
Total Posts:  1140
Joined:  2007-10-10
Paris, France
 

Have you seen this post ?
Can you try adding only

AddDefaultCharset Off
in your .htaccess ?

Don’t use

AddDefaultCharset utf-8

It may tell apache not using any default charset. That would be a better solution.

If this works, it would be better to have it in Moshe’s modification (with note).

 
Magento Community Magento Community
Magento Community
Magento Community
 
Grayson
Member
 
Avatar
Total Posts:  58
Joined:  2007-11-20
Netherlands
 

[

AddDefaultCharset Off
That works for me equally well. Good idea, SeL. So the new .htaccess could contain this:

############################################
## Prevent character encoding issues from server overrides 
## If you still have problems, use the second line instead

AddDefaultCharset Off
#AddDefaultCharset UTF-8
 
Magento Community Magento Community
Magento Community
Magento Community
 
marciomaciel
Jr. Member
 
Avatar
Total Posts:  28
Joined:  2009-06-01
Brazil
 

I’m having this problem too. When searching for ‘bonés’ in the store and click the search button its works and the term ‘bonés’ url goes to the store like this: “http://www.esporteline.com.br/catalogsearch/result/?q=bonés”. If I go to the url and hit enter it shows the error and change the url to “http://www.esporteline.com.br/catalogsearch/result/?q=bon&#xE9;s” and generates this warning in my log and dont exec the search.

The problem is that google is indexing this terms with special characters, and the call for indexing generates the error.

I’ve tried to add the information in htaccess but did not have any effect ...

Any ideas?

(sorry my english.... I only speak portuguese)

 
Magento Community Magento Community
Magento Community
Magento Community
Magento Community
Magento Community
Back to top