What type of encoding does wikipedia use?

Question

I have the wikipedia data dump and trying to decode special characters in the page titles, except a lot of characters don't match up the "standard" ascii encoding (referencing from here.)

As an example, in wikipedia ë and ã are given as:

ë = %C3%AB

ã = %C3%A3

Is there a defined key anywhere I can pull from?

[Why is ë percent encoded to %EB but also to %C3%AB?](http://stackoverflow.com/questions/12860837/why-is-%C3%AB-percent-encoded-to-eb-but-also-to-c3ab) — Robert Harvey, Mar 22 '16 at 17:38
That's my question... in the data set it's listed as %C3%AB but not sure how to get back to standard ascii — AltusVultur, Mar 22 '16 at 17:40
ë *cannot* be represented in ASCII, which defines only characters 0 to 127. — Philip Kendall, Mar 22 '16 at 17:52

score 7 · Accepted Answer · 2016-03-22T18:09:48.840

7

It's UTF-8.

Besides, neither character is in ASCII. They're in various "extended ASCII" character sets, but these encodings are not ASCII, they're remnants of a wild west age of character encodings. Treat them as legacy encodings that civilized people like us may have to decode but ideally should never produce. At least for ASCII there is a single table which almost the entire western world can agree on (and the rest of the world if they use UTF-8), while "extended" character sets are so numerous that it's anyone's guess what any given byte above 127 means.

The page you're linking to tacitly assumes one of these many "extended" character sets and (if a quick search didn't betray me) fails to mention. Now, in English texts it's often safe to assume some variant of Latin-1 (or ISO-whatsthenumber etc.) is implied, but it's still sloppy. Furthermore, as far as I am aware, there is by no means any standard as to what encoding percent-encoded bytes should be interpreted as. Again Latin-1 etc. are common but far from universal even in English language text. You should really get better sources.

edited Mar 22 '16 at 18:09

answered Mar 22 '16 at 17:51

"..nor part of the civilized world?" O_o – Robert Harvey Mar 22 '16 at 18:02
@RobertHarvey clearer now? – Mar 22 '16 at 18:13
Certainly more politically correct, yes. – Robert Harvey Mar 22 '16 at 18:46
@RobertHarvey Now I'm curious: Did you read the earlier version as referring to the characters ë and ã? In restrospect could see it being ambiguous, which was really unfortunate because it's the opposite of what I meant. I hate these 8-bit encodings to a large degree because they gives English-only speakers who want to write Æ an excuse to not go straight to Unicode. Or is it politically incorrect to call antiquated inadequate character encodings "uncivilized" now? :P – Mar 22 '16 at 18:55
1

If it wasn't already clear, it is now. **tl;dr:** *Go straight to Unicode.* – Robert Harvey Mar 22 '16 at 19:00

What type of encoding does wikipedia use?

1 Answers1