問題描述

ICU 項目(現(xiàn)在也有一個 PHP 庫) 包含幫助規(guī)范化 UTF-8 字符串所需的類，以便在搜索時更容易比較值.

The ICU project (which also now has a PHP library) contains the classes needed to help normalize UTF-8 strings to make it easier to compare values when searching.

但是，我正在嘗試弄清楚這對應(yīng)用程序意味著什么.例如，在哪些情況下我需要規(guī)范等效"而不是兼容性等效"，或者反之亦然?

However, I'm trying to figure out what this means for applications. For example, in which cases do I want "Canonical Equivalence" instead of "Compatibility equivalence", or vis-versa?

推薦答案

關(guān)于 Unicode 規(guī)范化你從未想知道的一切

規(guī)范歸一化

Unicode 包括多種對某些字符進行編碼的方法，尤其是重音字符.規(guī)范歸一化將代碼點更改為規(guī)范編碼形式.除字體或渲染引擎中的任何錯誤外，生成的代碼點應(yīng)與原始代碼點相同.

Everything You Never Wanted to Know about Unicode Normalization

Canonical Normalization

Unicode includes multiple ways to encode some characters, most notably accented characters. Canonical normalization changes the code points into a canonical encoding form. The resulting code points should appear identical to the original ones barring any bugs in the fonts or rendering engine.

因為結(jié)果看起來相同，所以在存儲或顯示字符串之前對字符串應(yīng)用規(guī)范化始終是安全的，只要您能容忍結(jié)果與輸入不完全相同.

Because the results appear identical, it is always safe to apply canonical normalization to a string before storing or displaying it, as long as you can tolerate the result not being bit for bit identical to the input.

規(guī)范標準化有兩種形式:NFD 和 NFC.從可以在這兩種形式之間轉(zhuǎn)換而不會丟失的意義上說，這兩者是等效的.在 NFC 下比較兩個字符串將始終給出與在 NFD 下比較它們相同的結(jié)果.

Canonical normalization comes in 2 forms: NFD and NFC. The two are equivalent in the sense that one can convert between these two forms without loss. Comparing two strings under NFC will always give the same result as comparing them under NFD.

NFD 將字符完全展開.這是計算速度更快的歸一化形式，但會產(chǎn)生更多代碼點(即使用更多空間).

NFD has the characters fully expanded out. This is the faster normalization form to calculate, but the results in more code points (i.e. uses more space).

如果您只想比較兩個尚未標準化的字符串，這是首選的標準化形式，除非您知道需要兼容性標準化.

If you just want to compare two strings that are not already normalized, this is the preferred normalization form unless you know you need compatibility normalization.

NFC 會在運行 NFD 算法后盡可能重新組合代碼點.這需要更長的時間，但會產(chǎn)生更短的字符串.

NFC recombines code points when possible after running the NFD algorithm. This takes a little longer, but results in shorter strings.

Unicode 還包括許多真正不屬于的字符，但在遺留字符集中使用.Unicode 添加這些是為了允許將這些字符集中的文本作為 Unicode 處理，然后無損地轉(zhuǎn)換回來.

Unicode also includes many characters that really do not belong, but were used in legacy character sets. Unicode added these to allow text in those character sets to be processed as Unicode, and then be converted back without loss.

兼容性規(guī)范化將這些轉(zhuǎn)換為相應(yīng)的真實"序列.字符，并且還執(zhí)行規(guī)范歸一化.兼容性規(guī)范化的結(jié)果可能與原始結(jié)果不一致.

Compatibility normalization converts these to the corresponding sequence of "real" characters, and also performs canonical normalization. The results of compatibility normalization may not appear identical to the originals.

包含格式信息的字符將替換為不包含的字符.例如，字符 ? 被轉(zhuǎn)換為 9.其他不涉及格式差異.例如羅馬數(shù)字字符Ⅸ被轉(zhuǎn)換為常規(guī)字母IX.

Characters that include formatting information are replaced with ones that do not. For example the character ? gets converted to 9. Others don't involve formatting differences. For example the roman numeral character Ⅸ is converted to the regular letters IX.

顯然，一旦執(zhí)行了這種轉(zhuǎn)換，就不可能再無損地轉(zhuǎn)換回原始字符集.

Obviously, once this transformation has been performed, it is no longer possible to losslessly convert back to the original character set.

Unicode Consortium 建議將兼容性規(guī)范化考慮為 ToUpperCase 轉(zhuǎn)換.它在某些情況下可能有用，但您不應(yīng)隨意應(yīng)用.

The Unicode Consortium suggests thinking of compatibility normalization like a ToUpperCase transform. It is something that may be useful in some circumstances, but you should not just apply it willy-nilly.

一個很好的用例是搜索引擎，因為您可能希望搜索 9 以匹配 ?.

An excellent use case would be a search engine since you would probably want a search for 9 to match ?.

您可能不應(yīng)該做的一件事是向用戶顯示應(yīng)用兼容性規(guī)范化的結(jié)果.

One thing you should probably not do is display the result of applying compatibility normalization to the user.

兼容性規(guī)范化形式有NFKD和NFKC兩種形式.它們與 NFD 和 C 之間的關(guān)系相同.

Compatibility normalization form comes in two forms NFKD and NFKC. They have the same relationship as between NFD and C.

NFKC 中的任何字符串本質(zhì)上也在 NFC 中，對于 NFKD 和 NFD 也是如此.因此NFKD(x)=NFD(NFKC(x))，和NFKC(x)=NFC(NFKD(x))，等等

Any string in NFKC is inherently also in NFC, and the same for the NFKD and NFD. Thus NFKD(x)=NFD(NFKC(x)), and NFKC(x)=NFC(NFKD(x)), etc.

如果有疑問，請使用規(guī)范標準化.根據(jù)適用的空間/速度權(quán)衡選擇 NFC 或 NFD，或根據(jù)您要與之互操作的事物的要求選擇 NFC 或 NFD.

If in doubt, go with canonical normalization. Choose NFC or NFD based on the space/speed trade-off applicable, or based on what is required by something you are inter-operating with.

這篇關(guān)于什么是標準化的 UTF-8?的文章就介紹到這了，希望我們推薦的答案對大家有所幫助，也希望大家多多支持html5模板網(wǎng)！