Why exactly can't PHP have full unicode support?

Question

Everybody knows, that PHP has problems with Unicode. Version 6 is effectively abandoned, because of Unicode implementation difficulties. But I wonder if anyone knows what are the exact reasons? Architecture/design problems, performance concerns, community problems (I bet not), something other?

Answer 1

PHP as a language definitely can have it, but I think the problem is with compatibility with existing programs. Unicode support can break them in subtle ways, which is the most annoying kind of bug to have.

Currently most string-processing functions in PHP are "binary-safe", which means you can use them to process any file in any encoding as well as binary formats like image data, etc.

With addition of Unicode strings you'd have to be very careful not to mix Unicode strings with binary strings (pretty hard when your strings come from different sources and you never had to worry about it before). And you couldn't be ignorant about encodings any more (and lots of scripts are ignorant about this!)

Another hard, but solvable problem is random access in Unicode strings. Implementation of $string[$offset] changes from trivial to either very slow or little slow and very complex.

Also I think it was a mistake to choose UTF-16 as internal encoding for PHP. It has same problems as UTF-8 (variable width because of surrogate pairs) and inefficiency of UCS-2. Maybe they should scrap that and start again with UTF-8?

</speculation>

Answer 2

TLDR: many PHP libraries are just a thin layer over native C libraries that don't support unicode, or support it in ways that are incompatible with each other. Rectifying this situation is likely to introduce backward incompatible changes.

DISCLAIMER: as I've switched from PHP to Python (to never look back) a few years ago, my opinion is clearly biased.

I think PHP is a nice and clever hack. As a hack, it started unpretentious and grew somewhat chaotically from a bunch of sparse libraries - lacking a well thought and unified design (from the computer language theory perspective).

As said by Machiavelli, "he who has not first laid his foundations may be able with great ability to lay them afterwards, but they will be laid with trouble to the architect and danger to the building".

For a programming language, the more popular, the harder to change. That is why languages like C changes once every 10 years. For example, Python 3 made many backward incompatible changes, and it was not pretty. The unicode support in previous Python incarnations was already considered superior to the current state of affairs in PHP, but guess what: the most polemic changes in Python 3 are related to unicode handling. This rant from Armin Ronacher summarizes the frustration from a huge share of the Python community.

PHP being "the" ubiquitous web platform makes it victim of its own success. Bring unified support for unicode in PHP is inevitable, but will require a lot of blood, sweat and tears.

Answer 3

One of the primary reasons the old PHP 6 work was stopped was due to the internal complexity it brought and the amount of work to do, which barely anybody fully unerstood.

A bit of history: PHP 6's Unicode imlementation was designed by the need of a larger PHP user and tried to do Unicode "right". After some evaluation the primary designer of PHP's to-be-Unicode-support has choosen to add a new string type which internally is Utf-16 and to allow different encdings to be used in different places. So the code might be written in one encoding, the output might use a different encoding and "runtme operations" some other encoding. The reason for choosing UTF-16 was that the work should be based on the ICU livrary which uses UTF-16 and it was found that this encoding makes common string operations in a fast way while conversi between utf- and utf-16 is relatively cheap. So far so good.

Now the consequence of doing this is at foremost the introduction of a new string type. PHP's internal type system till then had a few types (NULL, bool, int/long, float/double, string, array, resource, object) and lots of code had some assumptions on this being the case. Besides such assumptions all functions operating on strings, and there are a lot of those, have to be evaluated individually and it has to be decided how to handle encodings. Should they work on binary strings or unicode strings? If a conversion is required which encoding should be used etc. and this is a lot of work and in some cases quite complicated to do right. Additionally the internal APIs became quite complicated, as most key APIs in PHP got versions for binary strings (the old one) and then often a version for "runtime encoded" strings, as well as utf-16 strings, creating quite a mess there ...

Over the process of doing that many developers stumbled over the coplexity, became annoyed by utf-16 and didn't like the fact that this would more than double memory usage and spend lots of time converting strings while breaking most existing applications. So, PHP being driven by volunteers, fewer and fewer developers were working on it and other things piled up and contributors became unhappy and in the end it had to be abandoned.

Now what might the future bring? - There is a slow evolution happening that more and more things in PHP ae built around utf-8. Not in a strong way with a custom type and forcing everything and currently the developrs aren't motivated to touch this hot iron. One can hope that somebody has a good proposal to make it work nicely, but currently "everybody" will run away if they only hear the word. :)

Answer 4

I guess the actual reason is that PHP developing team lacks a clear roadmap for PHP development (let's just mention a pretty heated discussion when someone on the php-internals decided to start PHP 5.4 branch without previously agreeing on what features 5.4 should contain). I like this language very much, but the way it's being developed makes me a bit worried.