首页 > 编程 > Python > 正文

利用Python将文本中的中英文分离方法

2020-01-04 14:10:23
字体:
来源:转载
供稿:网友

在进行文本分析、提取关键词时,新闻评论等文本通常是中英文及其他语言的混杂,若不加处理直接分析,结果往往差强人意。

下面对中英文文本进行分离做一下总结:

1、超短文本,ASCII识别。

s = "China's Legend Holdings will split its several business arms to go public on stock markets, the group's president Zhu Linan said on Tuesday.该集团总裁朱利安周二表示,中国联想控股将分拆其多个业务部门在股市上市。"result = "".join(i for i in s if ord(i) < 256)print(result)
out:China's Legend Holdings will split its several business arms to go public on stock markets, the group's president Zhu Linan said on Tuesday.

2、unicode编码识别

import res = "China's Legend Holdings will split its several business arms to go public on stock markets, the group's president Zhu Linan said on Tuesday.该集团总裁朱利安周二表示,中国联想控股将分拆其多个业务部门在股市上市。"uncn = re.compile(r'[/u0061-/u007a,/u0020]')en = "".join(uncn.findall(s.lower()))print(en)
out:chinas legend holdings will split its several business arms to go public on stock markets, the groups president zhu linan said on tuesday

中文的编码范围是:/u4e00-/u9fa5,相应的[^/u4e00-/u9fa5]可匹配非中文。

匹配英文时,需要将空格[/u0020]加入,不然单词之间没空格了。

以上这篇利用Python将文本中的中英文分离方法就是小编分享给大家的全部内容了,希望能给大家一个参考,也希望大家多多支持VEVB武林网。


注:相关教程知识阅读请移步到python教程频道。
发表评论 共有条评论
用户名: 密码:
验证码: 匿名发表