blocks|key|1142457|text|嗯，你可以用NLP方法构建一个分类器，然后用噪声和无噪声的例子来训练它。您可以接受的一个例子是Apache的语言检测器。如果语言检测器说“打败我”，那可能就足够好了。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1142458|entityMap^0|0^^$0|@$1|2|3|4|5|6|7|D|8|@]|9|@]|A|$]]|$1|B|3|-4|5|6|7|E|8|@]|9|@]|A|$]]]|C|$]]

Well, you can build a classifier using NLP methods, and train it on examples of noise and not-noise. One case of that you can take is the language detector from Apache Tika. If the language detector says 'beats me' that might be good enough.

blocks|key|1142512|text|移除这样的东西并不像看上去那么容易。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1142513|对我们人类来说，很容易看出"djkhfkjh“没有任何意义。但是，计算机如何检测到这种噪音呢？它怎么知道“Eyjafjallaj+kull”仅仅是有人砸了他的键盘，或者是过去几年里最喧闹的山？|1142514|如果没有很多假阳性，你就无法可靠地做到这一点，所以，它毕竟是在手动过滤假阳性和真阳性。|1142515|entityMap^0|0|0|0^^$0|@$1|2|3|4|5|6|7|H|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|I|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|J|8|@]|9|@]|A|$]]|$1|F|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|G|$]]

Removing stuff like that isn't as easy as it might seem.

For us humans, it's easy to see that "djkhfkjh" doesn't make any sense. But how would a computer detect this kind of noise? How would it know that "Eyjafjallajökull" is just someone smashing his keyboard, or the most overbuzzed mountain in the last couple of years?

You can't do this reliably without many false positives, so after all, it's filtering the false-positives and true-positives by hand again.

blocks|key|1141926|text|获取一个具有尽可能多的名称的字典，并过滤数据以显示字典中没有的名称。然后，您必须逐个删除它们，以确保不删除有效数据。按名称对列表进行排序可以帮助您一次删除更多行。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1141927|entityMap^0|0^^$0|@$1|2|3|4|5|6|7|D|8|@]|9|@]|A|$]]|$1|B|3|-4|5|6|7|E|8|@]|9|@]|A|$]]]|C|$]]

Get a dictionary with as many names you can find and filter your data to display the ones that are not in the dictionary. Then you have to delete them one by one to make sure you do not delete valid data.
Sorting the list by name can help you delete more rows at a time.

blocks|key|2344330|text|如果课文的其余部分是英语，你可以使用单词列表。如果超过给定百分比(例如，50%25)的文本中的单词不在单词列表中，则可能是噪音。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|2344331|您可能想设置一个阈值，比如说，5个单词，以防止删除像'LOL‘这样的帖子。|2344332|在大多数Linux安装中，您可以从拼写检查器aspell中提取单词列表，如下所示：|offset|length|style|CODE|2344333|aspell+--lang+en+dump+master|code-block|syntax|javascript|2344334|entityMap^0|0|0|M|6|0|0^^$0|@$1|2|3|4|5|6|7|Q|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|R|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|S|8|@$F|T|G|U|H|I]]|9|@]|A|$]]|$1|J|3|K|5|L|7|V|8|@]|9|@]|A|$M|N]]|$1|O|3|-4|5|6|7|W|8|@]|9|@]|A|$]]]|P|$]]

If the rest of the text is English, you could use a word list. If more than a given percentage (say, 50%) of the words in the text are not in the word list, it is probably noise.

You may want to set a threshold of, say, 5 words, to prevent deleting posts like 'LOL'.

On most Linux installations, you can extract a word list from the spell checker <code>aspell</code> like this:

<pre><code>aspell --lang en dump master
</code></pre>

blocks|key|2344361|text|您需要从更有效地定义“噪音文本”开始。定义问题是这里最困难的部分。您不能编写会说“去掉类似于_____的字符串”的代码。看起来，您所识别的模式是“一行中有三个字符组成的一致集合，并且至少重复一次，但可能不会完全终止(它可以在集合中间的一个字符上终止)。”|type|unstyled|depth|inlineStyleRanges|entityRanges|data|2344362|现在编写一个与该模式匹配的正则表达式，并对其进行测试。|2344363|但我敢打赌你还在寻找其他模式..。|2344364|entityMap^0|0|0|0^^$0|@$1|2|3|4|5|6|7|H|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|I|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|J|8|@]|9|@]|A|$]]|$1|F|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|G|$]]

You're going to need to start by defining "noise text" more effectively. Defining the problem is the hard part here. You can't write code that will say "get rid of strings that are sort of like _____." It looks like the pattern you've identified is "a consistent set of three characters in a row, and the set repeats at least once, but may not terminate cleanly (it could terminate on a character from the middle of the set)."

Now write a regular expression that matches that pattern, and test it.

But I bet there are other patterns that you're looking for...

blocks|key|2389076|text|检查每个单词，看看有多少冗余。如果有三个以上的连续重复字母组，这是一个很好的候选噪音。另外，查找通常不属于一起的字母组和键盘上也是连续的字母组。如果一个完整的单词是由键盘相邻的字母组成的，它也声称在噪音列表中有一个位置。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|2389077|entityMap^0|0^^$0|@$1|2|3|4|5|6|7|D|8|@]|9|@]|A|$]]|$1|B|3|-4|5|6|7|E|8|@]|9|@]|A|$]]]|C|$]]

Inspect each word and see how much redundancy is there. If there are more than three consecutive repeated groups of letters, it is a good candidate for noise. Also, look for groups of letters that don't usually belong together and for groups of consecutive letters that are also consecutive on the keyboard. If a whole word is made of such letters that are keyboard neighbors, it also claims a spot on the noise list.

blocks|key|1142105|text|训练NLP分类器可能是最好的方法。然而，一个更简单的方法可能是简单地检查每个单词是否存在于所有已知的“有效”单词的列表中。大多数Unix系统都有一个名为/usr/share/dict/word的文件，可以用于此目的。此外，Ubuntu用/usr/share/dict/american-english、/usr/share/dict/american和/usr/share/dict/american扩展了这一点，每个列表都比最后一个列表更全面。这些列表还包含了很多常见的拼写错误，所以你不会过滤掉那些在技术上不是一个单词，但很明显可以识别为一个单词的文本。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1142106|如果你真的很有野心，你可以结合这些方法，并使用这些单词列表来训练贝叶斯或最大熵分类器。|1142107|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|F|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|G|8|@]|9|@]|A|$]]|$1|D|3|-4|5|6|7|H|8|@]|9|@]|A|$]]]|E|$]]

Training a NLP classifier would probably be the best way to go. However, a simpler method might be to simply check that each word exists in a list of all known "valid" words. Most Unix systems have a file called /usr/share/dict/words that you can use for this purpose. Additionally, Ubuntu expands on this with /usr/share/dict/american-english, /usr/share/dict/american-huge, and /usr/share/dict/american-insane, each list more comprehensive then the last. These lists also include a lot of common misspellings, so you won't filter out text that's not technically a word, but clearly recognizable as a word.

If you're really ambitious, you can combine these approaches, and use these words lists to train a Bayesian or Maximum Entropy classifier.

blocks|key|2389277|text|这里有很多好的答案。哪一个对你有用，很大程度上取决于你问题的具体情况。例如，输入应该是英语单词，用户名，人的姓等等。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|2389278|一种方法是:编写一个程序来分析您认为“有效”的输入。记录每一个可能的三个字母顺序出现在合法文本中的频率。然后，当您需要检查输入时，请查看输入的每个三个字母序列，并查找其预期的频率。类似于"xzt“的频率可能接近于零。如果你有太多这样的子序列，把它标记为垃圾。|2389279|这方面的问题：|2389280|你可能会把糟糕的拼写当成垃圾，例如，如果某人忘记在单词中的'q‘后面加上'u’，你就不会像"thethethethe".那样捕捉输入。|2389281|entityMap^0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|J|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|K|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|L|8|@]|9|@]|A|$]]|$1|F|3|G|5|6|7|M|8|@]|9|@]|A|$]]|$1|H|3|-4|5|6|7|N|8|@]|9|@]|A|$]]]|I|$]]

There are a lot of good answers here. Which one(s) will work for you depends a lot on the specifics of your problem -- for example, is the input supposed to be English words, usernames, people's last names, etc.

One approach: write a program to analyze what you consider "valid" input. Keep track of how frequently every possible three-letter sequence appears in legitimate text. Then when you have input to check, look at each three-letter sequence of the input and look up its expected frequency. Something like "xzt" probably has a frequency near zero. If you have too many subsequences like that, mark it as garbage.

Problems with this:

<ol>
<li>You might treat bad spelling as garbage, for example if someone forgets to put a 'u' after a 'q' in a word.</li>
<li>You won't catch input like "thethethethe".</li>
</ol>

blocks|key|2344599|text|示例1和#2可以由一个解析器删除，该解析器试图找出如何发音文本。不管语言是什么，它们都是说不出的，因此不是语言。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|2344600|entityMap^0|0^^$0|@$1|2|3|4|5|6|7|D|8|@]|9|@]|A|$]]|$1|B|3|-4|5|6|7|E|8|@]|9|@]|A|$]]]|C|$]]

Examples #1 and #2 can be removed by a parser that tries to figure out how to pronounce the text. Regardless of language they're unspeakable and thus not words.

giving a database table with huge data in it, what is the best practice to remove noise text such as :

<ul>
<li>fghfghfghfg</li>
<li>qsdqsdqsd</li>
<li>rtyrtyrty</li>
</ul>

that noise is stored into the "name" field.

I'm working on data with Java standard structures.

Detect and remove noise text

翻译质量差，导致语言生硬或混乱。

没有提供实际的解决方法或示例。

解答不清晰，无法理解或解决问题。

页面排版不美观，阅读体验差。

文章

问答

视频

学习中心

腾讯云实验室

直播

竞赛

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋 

腾讯云代码助手

CODING DevOps

Cloud Studio

SDK中心

API中心

命令行工具

给出一个包含大量数据的数据库表，删除噪声文本的最佳做法是什么，如：fghfghfghfgqsdqsdqsdrtyrtyrty该噪音存储在“名称”字段中。我正在使用Java标准结构处理数据。

问检测和删除噪声文本
EN

回答 9

Stack Overflow用户

Stack Overflow用户

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问检测和删除噪声文本EN

回答 9

Stack Overflow用户

Stack Overflow用户

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问检测和删除噪声文本
EN