NAV 2009 and Unicode!

  • [アーティクル]

The title might be a bit misleading, but I am writing this post as a response to a problem, which a partner ran into with NAV 2009 - and the problem is caused by Unicode. I am not a Unicode expert, so bare with me if I am naming some things wrong.

As you know, NAV 2009 is a 3T architecture and the Service Tier is 95% managed code (only the lower pieces of the data stack is still unmanaged code). You probably also know, that managed code supports Unicode natively - in fact a string in C# or VB.Net is by default a Unicode string.

In C# you use byte[] if you need to work with binary data. My earlier post about transferring binary data between the Service Tier and Client Side COM or Web Service consumers (you can find it here) I read the file content into a byte[] and use a base64 encoding to encode this content into a string, which is transferable between platforms, code pages etc.

Is NAV 2009 then Unicode compliant?

No, we cannot claim that. There are a lot of things, that stops us from having a fully Unicode compliant product - but we are heading in that direction. Some of the things are:

  • The development environment and the classic client is not Unicode. When you write strings in AL code they are in OEM.
  • The import/export formats are OEM format.
  • I actually don't know what the format inside SQL is, but I assume it isn't Unicode (again because we have mixed platforms).

Lets take an example:

In a codeunit I write the following line:

txt := 'ÆØÅ are three Danish letters, ß is used in German and Greek.';

Now, I export this as a .txt file, and view this in my favorite DOS text viewer:

image

As you can see - the Ø has changed and a quick look at my codepage reveals that I am running 437 - the codepage that doesn't have an Ø.

Opening the exported file in Notepad looks different:

Txt := '’î? are three Danish letters, á is used in German and Greek.';

Primary reason for this is, that Notepad assumes Unicode and the .txt file is OEM. When I launch the RoleTailored Client, and take a look at that line in the generated C# source file in Notepad:

txt = new NavText(1024, @"ÆØÅ are three Danish letters, ß is used in German and Greek.");

Nice - and we know that Notepad is Unicode compliant and C# uses Unicode strings, so the AL -> C# compiler converts the string to Unicode - how does this look in my favorite DOS text viewer:

image

Clearly my string has been encoded.

But wait...

If NAV 2009 converts my text constants to Unicode - what if I write this string to a file using my well known file commands in AL? - well, let's try.

I added the following code to a Codeunit and ran it from both the Role Tailored Client and from the Classic Client.

file.CREATETEMPFILE();
filename := file.NAME;
file.CLOSE();
file.CREATE(filename);
file.CREATEOUTSTREAM(OutStr);
OutStr.WRITETEXT(Txt);
file.CLOSE();

and don't worry, the results are identical - both platforms actually writes the file in OEM format (we might have preferred Unicode, but for compatibility reasons this is probably a good thing).

Another thing we should try, is to call a managed COM object - and take a look at the string we get there - and again we get the same string from the Classic and from the Role Tailored Client - but in this case, we do NOT get the string in OEM format - we get a Unicode string.

MSXML

Now if we call an unmanaged COM object (like MSXML2 - XMLHTTP) we actually get the OEM string when invoked from the Classic Client and a Unicode string when invoked from the Role Tailored Client. Typically XMLHTTP is used with ASCII only - but in some cases, they do have binary data - which might be in the 128-255 character range.

Our problem now is, that our binary data (which didn't have any relation to any special characters) gets converted to Unicode - and the Web Service provider doesn't stand a chance to guess what we mean with the data we send.

The next problem is that the Role Tailored Client doesn't support a byte[] type (binary) - in which we could have build a command and send it over. I tried a number of things, but didn't find a way to send any binary data (above 128) to the Send command of XMLHTTP.

The third problem with XMLHTTP is that the only way we can get a result back is by reading the ResponseText - and that is treated as a Unicode string and gets crumbled before we get it into NAV.

Remember that these problems will not occur if the web service provider uses XML to transfer data back and forth - since XML is ASCII compliant.

My first proposal if you are having problems with a Web Service provider, using a binary communication is to query the provider and ask for an XML version of the gateway. If this isn't possible - you have a couple of options (which both include writing a new COM object).

Create a proxy

You could create a Web Service proxy as a COM component (probably server side) and have a higher level function you call. This would remove the XMLHTTP glue code from NAV and put that into the COM object.

Example - you have a Web Service provider who can verify credit card numbers - and normally you would build up a string in AL and send this to the Send command - and then parse the ResponseStream you get back to figure out whether everything was A OK for not.

Create a function in a new COM object which might be named:

int CheckCreditCard(string CreditCardNo, string NameOnCard, int ExpireMonth, int ExpireYear, int SecurityCode)

Then your business logic in AL would just call and check - without a lot of XML parsing and stuff.

This would be my preferred choice - but it does involve some refactoring and a new COM object that needs to be installed on the server.

Use temporary files to transfer data

As mentioned earlier - writing a file in NAV 2009 with binary data, creates a file in OEM format - which would be the same binary content as we are typing in the AL editor.

So, you could create the string you want to send to XMLHTTP in a temporary file, create a new COM object which contains a function which sends the content of a file to a XMLHTTP object and writes the response back into the same file for NAV to read.

The idea here is that files (and byte[]) are binary data - strings are not.

The function in the COM object could look like:

public int SendFileContent(object NAVxmlhttp, string filename)
{
MSXML2.ServerXMLHTTP xmlHttp = NAVxmlhttp as MSXML2.ServerXMLHTTP;
FileStream fs = File.OpenRead(filename);
BinaryReader br = new BinaryReader(fs);
int len = (int)fs.Length;
byte[] buffer = new byte[len];
br.Read(buffer, 0, len);
br.Close();
fs.Close();

    xmlHttp.send(buffer);

    buffer = xmlHttp.responseBody as byte[];
fs = File.Create(filename);
BinaryWriter bw = new BinaryWriter(fs);
bw.Write(buffer);
bw.Close();
fs.Close();

    return xmlHttp.status;
}

If you went for this approach your AL code would change from:

XmlHttp.Send(Txt);

(followed by opening the ResponseStream) to

file.CREATETEMPFILE();
filename := file.NAME;
file.CLOSE();
file.CREATE(filename);
file.CREATEOUTSTREAM(OutStr);
OutStr.WRITETEXT(Txt);
file.CLOSE();

myCOMobject.SendFileContent(XmlHttp, filename);

(followed by opening the file <filename> again - reading the result).

The second approach would also work from the Classic client - so you don't have to use IF ISSERVICETIER THEN to do the one of the other.

In the Edit In Excel - Part 4, you can see how to create a COM object - should be pretty straightforward.

Build a binary string that doesn't get encoded

You could create a function in a COM object, which returns a character based on a numeric value:

public string GetChar(int ch)
{
return ""+(char)ch;
}

Problem with this direction is, that this function should ONLY be used when running in the Role Tailored Client.

Calling this function from the Classic Client will case this string to be seen as Unicode and converted back into OEM - and that really wouldn't make sense at all.

Convert to/from Unicode using strings instead of files

So what if the data you need to send is confidential and you cannot write that to a file?

Well - you can create a function, which converts the string back to OEM (making it the same binary image) - send it over the wire - and then convert the response to UniCode (so that when the string comes into NAV - it will be converted back to OEM again again).

Seems like a lot of conversion back and forth - but it would actually work from both the Classic and the Role Tailored Client, the code for that goes here:

public class MyCOMobject : IMyCOMobject
{
private static Byte[] oem2AnsiTable;
private static Byte[] ansi2OemTable;

    /// <summary>
/// Initialize COM object
/// </summary>
public MyCOMobject()
{
oem2AnsiTable = new Byte[256];
ansi2OemTable = new Byte[256];
for (Int32 i = 0; i < 256; i++)
{
oem2AnsiTable[i] = (Byte)i;
ansi2OemTable[i] = (Byte)i;
}
NativeMethods.OemToCharBuff(oem2AnsiTable, oem2AnsiTable, oem2AnsiTable.Length);
NativeMethods.CharToOemBuff(ansi2OemTable, ansi2OemTable, ansi2OemTable.Length);
// Remove "holes" in the convertion structure
Int32 ch1 = 255;
Int32 ch2 = 255;
for (;; ch1--, ch2--)
{
while (ansi2OemTable[oem2AnsiTable[ch1]] == ch1)
{
if (ch1 == 0)
break;
else
ch1--;
}
while (oem2AnsiTable[ansi2OemTable[ch2]] == ch2)
{
if (ch2 == 0)
break;
else
ch2--;
}
if (ch1 == 0)
break;
oem2AnsiTable[ch1] = (Byte)ch2;
ansi2OemTable[ch2] = (Byte)ch1;
}
}

    /// <summary>
/// Convert Unicode string to OEM string
/// </summary>
/// <param name="str">Unicode string</param>
/// <returns>OEM string</returns>
private byte[] UnicodeToOem(string str)
{
Byte[] buffer = Encoding.Default.GetBytes(str);
for (Int32 i = 0; i < buffer.Length; i++)
{
buffer[i] = ansi2OemTable[buffer[i]];
}
return buffer;
}

    /// <summary>
/// Convert OEM string to Unicode string
/// </summary>
/// <param name="oem">OEM string</param>
/// <returns>Unicode string</returns>
private string OemToUnicode(byte[] oem)
{
for (Int32 i = 0; i < oem.Length; i++)
{
oem[i] = oem2AnsiTable[oem[i]];
}
return Encoding.Default.GetString(oem);
}

    /// <summary>
/// Send data through XMLHTTP
/// </summary>
/// <param name="NAVxmlhttp">XmlHttp object</param>
/// <param name="data">string containing data (in Unicode)</param>
/// <returns>The response from the XMLHTTP Send</returns>
public string Send(object NAVxmlhttp, string data)
{
MSXML2.ServerXMLHTTP xmlHttp = NAVxmlhttp as MSXML2.ServerXMLHTTP;
byte[] oem = UnicodeToOem(data);
xmlHttp.send(oem);
        return OemToUnicode((byte[])xmlHttp.responseBody);
}
}

internal static partial class NativeMethods
{
#region Windows OemToChar/CharToOem imports

    [DllImport("user32", EntryPoint = "OemToCharBuffA")]
internal static extern Int32 OemToCharBuff(Byte[] source, Byte[] dest, Int32 bytesize);

    [DllImport("user32", EntryPoint = "CharToOemBuffA")]
internal static extern Int32 CharToOemBuff(Byte[] source, Byte[] dest, Int32 bytesize);

    #endregion
}

Unfortunately I have not found a way to do this without having a COM object in play.

Enjoy

Freddy Kristiansen PM Architect
Microsoft Dynamics NAV

Comments

  • Anonymous
    February 06, 2009
    Hi Freddy, Thank you for a great post. You write: "No, we cannot claim that. There are a lot of things, that stops us from having a fully Unicode compliant product - but we are heading in that direction." How far away from a real Unicode solution are we? IMHO then if they guys at MSFT who makes the decisions are serious about NAV, then they should do something about this problem. Erik

  • Anonymous
    February 06, 2009
    I think we are doing something about this problem - but it is all about priority. I don't think you will see NAV being Unicode before we get rid of the Classic Client and go full 3T only - the amount of work, that we need to put into the Classic Client in order to make that support Unicode is huge and this is the reason why it hasn't been done.

  • Anonymous
    February 19, 2009
    It is possible to accomplish the conversion from ansi to unicode and visa versus, from within NAV classic. When using ADO Stream automation, it is possible to convert NAV ANSI strings and files with C/AL code, to whatever encoding you like.  

  • Anonymous
    February 20, 2009
    That is correct - but beware that this conversion is destructive (which is the reason for a lot of the code in this post). This post isn't so much about converting a string to unicode - as it is explaining how the internals of the NAV 2009 service tier handles strings and converting back and forth. For that conversion we of course need the conversion to be non-destructive.

  • Anonymous
    May 10, 2009
    Hi, I have published a codeunit with a function(name: Transfer(bigtext)) that have a BigText as parameter. If a use the webserservice i WS like: Transfer(xmlFile); Where XmlFile is a XmlFile converted to a string. If the XmlFile contains characters like Æ,Ø,Å these are not correct formatted in NAV. in NAV they look like: ‘,›,† Do you have a surggestion on how to solve this challange??

  • Anonymous
    May 10, 2009
    I guess you would need to convert it - but you need to tell me some more... You read a file in a C# app and call into NAV with the file content. What character set is the file? Did you validate that the file is actually Unicode when it is in your C# app?

  • Anonymous
    May 10, 2009
    Correct -> You read a file in a C# app and call into NAV with the file content. the character set of the file is utf-16 (<?xml version="1.0" encoding="utf-16" ?> ) Yes the file is unicode. Even when you define a string as: string text = "æøå"; This will not be formatted correct in NAV when used as parameter. I also tried to encode the string as base64 and decode it again in NAV. With the same result. BUT when I decoded the string and put it into a table(text field) then it look okay(æøå). But I put it into a instream and send this to xmlport... then not formatted correct

  • Anonymous
    May 11, 2009
    You will need to show some sample code here in order for me to understand the problem better. I tried creating two different functions: Test1(str : Text[40]) result : Text[40] result := str; Test2(obj : BigText) result : Text[40] obj.GETSUBTEXT(str, 1); result := str; And call these from WS with æøå as parameter - and in both cases, I get æøå back, meaning that in this simple situation - it will be converted correctly back and forth from Ansi to Unicode.

  • Anonymous
    May 11, 2009
    Here is a senario where I write to a file: A/L code: WriteToFile(VAR big : BigText) _File.WRITEMODE(TRUE); IF _File.OPEN('c:output.txt')  THEN  _File.TRUNC ELSE  _File.CREATE('c:output.txt'); _File.CREATEOUTSTREAM(_OutStream); _File.CREATEINSTREAM(_InStream); big.WRITE(_OutStream); _File.CLOSE; In WS the code look like tis: TestService testService = new TestService.TestService();            testService.UseDefaultCredentials = true;            try            {                string s = "æøå";                testService.WriteToFile(ref s);                      }            catch (Exception ex)            {            } I get this output in my file(c:output.txt): ‘›† It must be the streams that c

  • Anonymous
    May 11, 2009
    What program are you using to look at the file with? Notepad? Notepad assumes that your .txt file is Unicode and all files written from NAV are OEM - go to a DOS prompt and type the file and you might see the æ and the å - the ø might be something different. Reason for this is compatibility - and in fact if you write a codeunit: _File.WRITEMODE(TRUE); IF _File.OPEN(TEMPORARYPATH + 'output.txt')  THEN _File.TRUNC ELSE _File.CREATE(TEMPORARYPATH + 'output.txt'); _File.CREATEOUTSTREAM(_OutStream); _File.CREATEINSTREAM(_InStream); str := 'æøå'; _OutStream.WRITETEXT(str); _File.CLOSE; and run this from the classic client, you will get the same result - ‘›† in notepad If you indeed want to write the file in Unicode I guess you need to make a convert to ansi (the code above) and then write it (which implicit will convert back to OEM (which then will be ansi - get it?)

  • Anonymous
    May 11, 2009
    Yes I use notepad to look at the file.... I think I understand what you writing but I cannot manage to get it working.. Do you have an exsample of how to send a unicode string into NAV? Yes I read the article above but you cannot send a byte[] to a published codeunit as parameter..

  • Anonymous
    February 24, 2011
    Hi. I am not able to copy your Sources. Missing Using Statesments in Visual Studio etc. Is it posible to post complete Sources?

  • Anonymous
    March 15, 2011
    I have a similar problem to Tennek.  I have a web service that takes a BigText parameter (so that the xml can be logged).  When it hits NAV it's been converted from umicode but some of the characters are incorrect.  For example ä becomes ,, (one character). The site sends in unicode and I can reproduce with a .NET program that reads the xml from a file. This result in the xmlport giving an error about illegal characters. The audit log (blob) can be exported and shown in notepad but this shows the incorrect character. I think its down to the conversion that NAV does and the code page of the server. I'm guessing that if I'd used the XMLPort as a parameter it would work but have not found a way to save the xml audit that way without writing a proxy. Any ideas would be very welcome.

  • Anonymous
    April 16, 2012
    Hello, I have similar problem with my web service too. In my case I have a function from a codeunit published as FunctionName(Param1: BigText; VAR Param2: BigText). In this case, I'm using Dynamics NAV 2009 R2 and its new variable type DotNet. I have used one .Net assembly to solve the problem of saving special characters into the database. But now my problem is when i retrieve information from the database through the BigText VAR param2, I keep getting weird characters instead of the ones that are expected. I have tried to use the DotNet assemblies to change the encoding of the text sent back through the BigText parameter, but still no luck. I have been debugging the Dynamics Nav Server and I found that the stream that gets loaded using a XMLPort, to return the data back through the BigText VAR Param2, has already the wrong characters before returning the value. Has anyone run into this problem, and been able to solve it in any way with the last version of Dynamics NAV? I would like to solve the problem without having to change the design of the web service function I have published. Any help would be really appreciated.

  • Anonymous
    May 15, 2013
    Hello, I got solution for Table Fields Unicode Caption. You can find it S#001 at my SkyDrive link skydrive.live.com/redir