CFMX 6.1 and charset handling problems

Ok, we started transferring our servers from CF5 to CFMX 6.1 couple of weeks ago and have otherwise had relatively easy migration, but some weird, weird problems with charset handling keep bugging us. This is not anything that we've been able to solve by using cfprocessingdirective or cfcontent or setEncoding().

The background:

CFMX6.1 running on RH Enterprise server, JRE1.4.2 with Apache 2.x. Database is mySQL sitting on another box, using latin1 charset encoding (can't be changed to UTF-8 as other servers access this data as well).

Our site has now this stuff (among others of course) in Application.cfm:

cfcontent for ISO-8859-1
setEncoding() for ISO-8859-1

also the website's HTML headers state ISO-8859-1

Now, usually everything goes just nicely, latin1 data is pulled from the db and shown on the site (I must also mention that data has "special characters", such as ä and ö, since the site is in Finnish) and special chars, etc are shown correctly.

Problems? Tons. CFMAIL and CFFILE being the worst.

CFFILE?

Pulling data from latin1 db and putting it into variable (<cfset textoutput="some static text"&query.column&" more static text"> and writing it to a new file simply doesn't work if there are special characters. Not even with <cffile charset="ISO-8859-1">. UTF-8 charset seems to work (but that has to be specified as well, with no charset the CFFILE outputs garbage), but its in UTF-8 then and we need everything to be in ISO-8859-1 on our site. Typically 'ä's are converted to either

a) 'y's with apostrophe on top of them (ANSI code 254)
b) question marks
c) unrendeable "boxes"

Hack?

Yep. Managed to hack past this, by downloading a custom tag called "charsetconvert" from MM's devex and modifying it a bit. It allows doing either UTF2ISO or ISO2UTF conversions.

How?

This is the interesting part:

<cfset variable="static text"&query.column&"more static text">

<cf_charsetconvert type="iso2utf" input="#variable#" output_var_name="variable_2">
<cf_charsetconvert type="utf2iso" input="#variable_2#" output_var_name="variable">

<cffile action="write" file="blah" output="#variable#">

works..

CFMAIL?

Same applies to CFMAIL, we need to do ISO2UTF and then UTF2ISO before we put the variable within <cfmail> tag's body.

Anything else?

Yep. Turn on the debug in CF administration and the whole site switches to use UTF-8 despite you setting it to use ISO-8859-1 in your files.

If your page has _anywhere_ a function DateFormat() or TimeFormat(), the whole page will be handled in UTF-8, not in ISO-8859-1. We had to create a custom function called "niceDate" that does exactly the same as DateFormat() and TimeFormat() do, but using just Month(), etc functions and replace all the DateFormat and TimeFormat functions with that to get it working.

Weirdest thing is this: Occassionally, maybe appx. every 10-30 hours, server switches itself into "hanging mode" (i.e. when you stop the CF server, it tells you "CFMX seems to be hanging, stopping non-gracefully" or something like that, where it works, but has forced all the pages to use UTF-8 in internal processing, when even the above-mentioned hacks for CFFILE and CFMAIL fail (and removing the hacks in this occassion wont work either). Only solution is to restart the server, after which, the hacks start working again.


Now.. Anyone has any idea whether this is caused by the underlying JRE version or something else that could be fixed by us and not have to wait for MM to release a patch for this? Or has anyone else experienced similar problems -- we have two identical web servers and both have this problem (apart from the "hanging" problem, the other box doesn't seem to hang and thus doesn't cause the occassional change in behaviour)?

Unfortunately I can't be of any help here. Mainly because am I not using international formatting so I haven't messed with it personally. But I frequent many lists and blogs on CFMX and I haven't heard anything like this before. My first instinct would be to consider it an application issue than a CFMX issue simply because no one else I've seen has had this problem.

Perhaps try the CF forums at Macromedia? Maybe one of the CF tech support guys can try to give a more detailed answer, or at least an idea of what the problem is.

I've also had problems with coldfusion code breaking in CF6.1 due to data with "special characters" (although the code worked in CF4.5)

But our "special characters" were nothing more than types of dashs and apostrophes (CHR 92, CHR 96 and CHR 97).
Not only is CFFILE affected, but CFOUTPUT breaks too: only question marks and unrenderable "boxes" are displayed.

CFOUTPUT was easily fixed by placing the following in application.cfm:
<cfset setEncoding("url","ISO-8859-1")>
<cfset setEncoding("form","ISO-8859-1")>
<cfcontent type="text/html; charset=ISO-8859-1">

I've also seen it recommend in other forums (although it has not helped in my instance) that at the top of each cfm page you should add:
<cfprocessingdirective pageEncoding="ISO-8859-1">
and in the advanced option, connection string, for the DSN you should add:
useUnicode=true&characterEncoding=ISO-8859-1

I'll try your fix for CFFILE...

I'm dealing with the issue right now and have found out that CF change their server charset to UTF-8. I have yet hear how to change the default server charset. Right now any page that CF processes will change the HTTP header to use the UTF-8 charset. It seems to overwrite or disable the webserver default charset or any metatags that set the charset.

If anyone finds an answer to this issue please post back to thread.

many many many problems with special chars after upgrade to mx....

some could be solved with inserting this code at the top of every page (or in application)

<cfcontent type="text/html; charset=ISO-8859-1">
<cfscript>
setencoding("form", "ISO-8859-1");
setencoding("url", "ISO-8859-1");
</cfscript>
<cfprocessingdirective pageencoding="iso-8859-1">

and this in the header of the html code

<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">

more helpfull tips could be found here:

http://piet.niederhausen.net/resources/character_sets.cfm

http://www.prokofiev.org/forum/viewmessages.cfm?Forum=2&Topic=704

http://www.dialoguedesigns.co.uk/dc2/page.cfm?&pageid=cfmxproblems&reload=0.69223804

for the moment i'm still strugeling with the DateFormat,... problems

greets KoenVH
http://www.d-arts.be

I feel your pain -- I created a CF function called "niceDate()" to replace the dateformat / timeformat issues (using either of them permanently switched CFMX into bizarre mode as described that refused to change the internal char encoding until restarted), so here it is in case you might find use for it:



Not to dismiss Macromedia's product, but due the constant char encoding problems and issues, specially under Linux, we switched to BlueDragon [free edition] in March and haven't considered going back to MX.. But hopefully the above code at least eases your pain slightly :-)

don't shoot the messenger and thanx for the function,

but i replaced DateFormat with LSDateFormat in my code, and (till now) it seems to be working...

on to the next bug ...

greets KoenVH
http://www.d-arts.be

and ps: after i debugged all my clients site's, i also think of saying goodbye to coldfusion... MX sucks bigtime

Like said, I hate bashing any software, but the switch to BD was right for us -- stability (of appx 20M page imps per month site) has improved radically (and gained without re-doing the code in some other language). And the most important thing that changed was that now we are communicating with a company that responds to bug reports within 12 hours (normally by their CTO :-) and actually do fix them typically within 1-3 months after its been confirmed.

Mailing to MM about possible bugs was as useful as sending your emails to /dev/null..

I've never had any problems getting help form the CF team when I have issues. Are you using the proper form, or have you already used up all of your complimentary support incidents?

BlueDragon is a good implementation of CFML and will work for most users, but there are many esoteric bugs and issues when you get into complex or advanced code. It is also always a full version behind CFMX. But if you can live with those two things it may be a nice alternative.

The problem then is almost certainly with your application. We migrated a huge (100,000+ lines of code) application from CF 5 to CFMX 6, to CFMX 6.1 to CFMX 7.0 and didn't have to change anything in any of the migrations.

kiteless: The problems with character encoding problems with MX and MX 6.1 were quite well detailed and confirmed by several members on Macromedia's site, as were the constant problems with CFMX6.1 simply shutting itself down on Linux environment (Java Signal 10 if I remember correctly..).

Both problems got either no help from CF team members ("Sorry, we use Windows..") or simply ignored by MM (submitted an official bug report to them via their bug submission form that I never got a reply to, even tho I after 3 months in waiting, emailed them again).

And I agree that BD is always one step behind and also has its own funky bugs -- the first issue can be ignored unless you always want to take full advantage of all the new features of the latest version (its bit like updating to latest MS Office -- everybody does it, but nobody can really in reality tell what of the new features they are using..), second one is compensated by the fact that NA at least confirms the bugs as a company and fixes them, typically within reasonable timeline.

About CF5 on Linux, didn't have much problems and I could still suggest people willing to run CFML code on Linux to use that environment -- for CF7, I don't have experience with -- but as far as CFMX / MX6.1 on Linux goes, it can be a smooth ride, but for us and several clients of ours, it proved to be nightmare. The whole episode could've been avoided by simply MM stepping in, stating "yes, we confirm this bug and we'll release a hotfix for in mm/dd/yyyy" or "we found the problem and you can get over it by doing <this>".

But enough of this -- I've used CF since v3.0 and love the product, so my comments are NOT meant to dismiss Coldfusion as an entity, just to highlight this particular problem described in this thread (which I also hope that CF7 has taken care of, so no new users of CF have to deal with it and the whole thread will fade into Net's "dead threads" archive :-).

Just out of curiosity, did you ever actually open a support incident?

Yes, I did.

And even the engineers who wrote the software could not determine the problem? It must have been an extraordinarily tenacious bug. Hopefully they did get it sorted out in CF7.

Like I said -- got nothing back from Macromedia regarding the bug submission. Investigated it 2 months myself and tracked it most likely down to the Sun's Java, as slightly similar issues were a concern among Java coders. But as you said, lets hope its been handled in CF7. Anyway, its past for us, just wanted to lend a hand to the poor guy struggling with the same issue -- we build ourselves an enormous hack system to get around the character encoding problems and the avoidance of DateFormat() and TimeFormat() by creating a similar replacement function was one of the pieces in the puzzle. Eventually the burden got simply slightly overwhelming to make the code in a way that it avoided this and that, in order to make the char encoding to stay stable at least a day at a time :-)