本文共 6258 字，大约阅读时间需要 20 分钟。
本文翻译自：
In our application, we receive text files ( .txt , .csv , etc.) from diverse sources. 在我们的应用程序中，我们从各种来源接收文本文件（ .txt ， .csv等）。 When reading, these files sometimes contain garbage, because the files where created in a different/unknown codepage. 读取时，这些文件有时包含垃圾，因为这些文件是在不同/未知的代码页中创建的。
Is there a way to (automatically) detect the codepage of a text file? 有没有办法（自动）检测文本文件的代码页？
The detectEncodingFromByteOrderMarks , on the StreamReader constructor, works for UTF8 and other unicode marked files, but I'm looking for a way to detect code pages, like ibm850 , windows1252 . 该detectEncodingFromByteOrderMarks ，对StreamReader构造，适用于UTF8等统一标记的文件，但是我正在寻找一种方法来检测代码页，像ibm850 ， windows1252 。
Thanks for your answers, this is what I've done. 感谢您的回答，这就是我所做的。
The files we receive are from end-users, they do not have a clue about codepages. 我们收到的文件来自最终用户，他们不了解代码页。 The receivers are also end-users, by now this is what they know about codepages: Codepages exist, and are annoying. 接收者也是最终用户，到目前为止，这是他们对代码页的了解：代码页存在并且令人讨厌。
Solution: 解：
参考：
I use this code to detect Unicode and windows default ansi codepage when reading a file. 读取文件时，我使用此代码检测Unicode和Windows默认的ansi代码页。 For other codings a check of content is necessary, manually or by programming. 对于其他编码，必须手动或通过编程检查内容。 This can de used to save the text with the same encoding as when it was opened. 这可以用来以与打开时相同的编码保存文本。 (I use VB.NET) （我使用VB.NET）
'Works for Default and unicode (auto detect)Dim mystreamreader As New StreamReader(LocalFileName, Encoding.Default) MyEditTextBox.Text = mystreamreader.ReadToEnd()Debug.Print(mystreamreader.CurrentEncoding.CodePage) 'Autodetected encodingmystreamreader.Close()
Have you tried 您是否尝试过
Example from 来自示例
public static void Main(String[] args){ string filename = args[0]; using (FileStream fs = File.OpenRead(filename)) { Ude.CharsetDetector cdet = new Ude.CharsetDetector(); cdet.Feed(fs); cdet.DataEnd(); if (cdet.Charset != null) { Console.WriteLine("Charset: {0}, confidence: {1}", cdet.Charset, cdet.Confidence); } else { Console.WriteLine("Detection failed."); } }} If you can link to a C library, you can use libenca . 如果可以链接到C库，则可以使用libenca 。 See . 请参阅 。 From the man page: 从手册页：
Enca reads given text files, or standard input when none are given, and uses knowledge about their language (must be supported by you) and a mixture of parsing, statistical analysis, guessing and black magic to determine their encodings. Enca会读取给定的文本文件，或者在没有输入文件的情况下读取标准输入，并使用有关其语言的知识（必须得到您的支持）以及解析，统计分析，猜测和黑魔法的组合来确定其编码。
It's GPL v2. 这是GPL v2。
Looking for different solution, I found that 寻找不同的解决方案，我发现
this solution is kinda heavy. 这个解决方案有点沉重。
I needed some basic encoding detection, based on 4 first bytes and probably xml charset detection - so I've took some sample source code from internet and added slightly modified version of 我需要一些基本的编码检测（基于前四个字节）和xml charset检测-因此，我从Internet上获取了一些示例源代码，并添加了经过稍微修改的
written for Java. 为Java编写。
public static Encoding DetectEncoding(byte[] fileContent) { if (fileContent == null) throw new ArgumentNullException(); if (fileContent.Length < 2) return Encoding.ASCII; // Default fallback if (fileContent[0] == 0xff && fileContent[1] == 0xfe && (fileContent.Length < 4 || fileContent[2] != 0 || fileContent[3] != 0 ) ) return Encoding.Unicode; if (fileContent[0] == 0xfe && fileContent[1] == 0xff ) return Encoding.BigEndianUnicode; if (fileContent.Length < 3) return null; if (fileContent[0] == 0xef && fileContent[1] == 0xbb && fileContent[2] == 0xbf) return Encoding.UTF8; if (fileContent[0] == 0x2b && fileContent[1] == 0x2f && fileContent[2] == 0x76) return Encoding.UTF7; if (fileContent.Length < 4) return null; if (fileContent[0] == 0xff && fileContent[1] == 0xfe && fileContent[2] == 0 && fileContent[3] == 0) return Encoding.UTF32; if (fileContent[0] == 0 && fileContent[1] == 0 && fileContent[2] == 0xfe && fileContent[3] == 0xff) return Encoding.GetEncoding(12001); String probe; int len = fileContent.Length; if( fileContent.Length >= 128 ) len = 128; probe = Encoding.ASCII.GetString(fileContent, 0, len); MatchCollection mc = Regex.Matches(probe, "^<\\?xml[^<>]*encoding[ \\t\\n\\r]?=[\\t\\n\\r]?['\"]([A-Za-z]([A-Za-z0-9._]|-)*)", RegexOptions.Singleline); // Add '[0].Groups[1].Value' to the end to test regex if( mc.Count == 1 && mc[0].Groups.Count >= 2 ) { // Typically picks up 'UTF-8' string Encoding enc = null; try { enc = Encoding.GetEncoding( mc[0].Groups[1].Value ); }catch (Exception ) { } if( enc != null ) return enc; } return Encoding.ASCII; // Default fallback } It's enough to read probably first 1024 bytes from file, but I'm loading whole file. 足以从文件中读取前1024个字节，但是我正在加载整个文件。
The tool "uchardet" does this well using character frequency distribution models for each charset. 工具“ uchardet”使用每个字符集的字符频率分布模型可以很好地完成此任务。 Larger files and more "typical" files have more confidence (obviously). 更大的文件和更多的“典型”文件具有更大的信心（显然）。
On ubuntu, you just apt-get install uchardet . 在ubuntu上，您可以apt-get install uchardet 。
On other systems, get the source, usage & docs here: 在其他系统上，请在此处获取源代码，用法和文档： :
转载地址：http://hwjnb.baihongyu.com/