Encoding non english characters may cause many problems. Especially if you are using a smart text editor, it’s hard to find out what is really happening in the communication because smart text editors are extremely cool moreover they fix the encoding errors by modifying the retrieved message in order to help developers. But sometimes this is not a help but a difficulty. Anyway we all love smart text editors and this is not about them, just about encoding conversions.

In this article, I would like to explain an issue I faced while developing remote services for SAP in order to communicate with an electronic store.
Here is the simplified diagram of the communication.

As you see remote site posts an XML package with CURL and Php Service receives the message and runs some query in the SAP database.

In order to show non-english characters SAP database was configured for ISO Latin 5 encoding
and I was receiving the xml post in UTF-8 format. When I print the incoming xml post, I haven’t faced with any trouble it seems like everything is correct but when I inserted to db, I always get character errors.

Then we find out that, textmate is configured to use default utf-8 format so although I receive different encoded xml post, textmate is showing it correct.

In order to regenerate what happened you should follow these steps.
- Open TextEdit and write non-english characters like: “ÇçĞğŞşİıIiÖöÜü”
- While saving the file choose iso latin 5 encoding and save.
- Now open file in textmate you will commonly see “ÇçÐðÞþÝýIiÖöÜü”

what we have done to solve this problem is simply replacing the mistaken characters with correct ones.
In this article I wanted to warn you to be careful about 4 major point,

1) which encoding type you are working on,
2) which encoding type your text editor using
3) which encoding type your database is storing
4) which encoding type other application that’s sharing the same database using.

If you face that kind of encoding problems, ask yourself those 4 questions. :)
if all of those 4 is in harmony, play the service ;)
I hope it will help
cheers.
Guvener

If you enjoyed this post, make sure you subscribe to my RSS feed!