最新消息:20190529 VPS服务器已从腾讯云香港换为Vultr新加坡,主题仍用朋友推荐的大前端D8

【已解决】Python脚本运行出错:libs/thirdparty\chardet\universaldetector.py:90: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode – interpreting them as being unequal

Python crifan 1989浏览 0评论

【问题】

在折腾自己的一个python脚本,把所使用的chardet库,从1.0.1升级到1.1后,结果运行我的的python脚本,却出错:

LINE 810  : INFO     [0001] http://againinput4.blog.163.com/blog/static/172799491201091513711591
LINE 886  : INFO       Title = intro
libs/thirdparty\chardet\universaldetector.py:90: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - inte
rpreting them as being unequal
  if aBuf[:len(chunk)] == chunk:
LINE 1617 : INFO     Exporting items at last ...

【解决过程】

1.后来换为旧的1.0.1版本的chardet,就可以正常运行了,就没了此警告。

2。去比较了一下,两者之间的代码的区别:

(1)1.0.1的chardet的universaldetector.py的相关的代码:

    def feed(self, aBuf):
        if self.done: return

        charmap = (
            # EF BB BF  UTF-8 with BOM
            ('\xEF\xBB\xBF', {'encoding': "UTF-8", 'confidence': 1.0}),
            # FF FE 00 00  UTF-32, little-endian BOM
            ('\xFF\xFE\x00\x00', {'encoding': "UTF-32LE", 'confidence': 1.0}),
            # 00 00 FE FF  UTF-32, big-endian BOM
            ('\x00\x00\xFE\xFF', {'encoding': "UTF-32BE", 'confidence': 1.0}),
            # FE FF 00 00  UCS-4, unusual octet order BOM (3412)
            (u'\xFE\xFF\x00\x00', {'encoding': "X-ISO-10646-UCS-4-3412", 'confidence': 1.0}),
            # 00 00 FF FE  UCS-4, unusual octet order BOM (2143)
            (u'\x00\x00\xFF\xFE', {'encoding': "X-ISO-10646-UCS-4-2143", 'confidence': 1.0}),
            # FF FE  UTF-16, little endian BOM
            ('\xFF\xFE', {'encoding': "UTF-16LE", 'confidence': 1.0}),
            # FE FF  UTF-16, big endian BOM
            ('\xFE\xFF', {'encoding': "UTF-16BE", 'confidence': 1.0}),
        )

        aLen = len(aBuf)
        if not aLen: return

        if not self._mGotData:
            # If the data starts with BOM, we know it is UTF
            for chunk, result in charmap:
                if aBuf[:len(chunk)] == chunk:
                    self.result = result
                    break

(2)1.1的chardet的universaldetector.py的相关的代码:

    def feed(self, aBuf):
        if self.done: return

        aLen = len(aBuf)
        if not aLen: return
        
        if not self._mGotData:
            # If the data starts with BOM, we know it is UTF
            if aBuf[:3] == '\xEF\xBB\xBF':
                # EF BB BF  UTF-8 with BOM
                self.result = {'encoding': "UTF-8", 'confidence': 1.0}
            elif aBuf[:4] == '\xFF\xFE\x00\x00':
                # FF FE 00 00  UTF-32, little-endian BOM
                self.result = {'encoding': "UTF-32LE", 'confidence': 1.0}
            elif aBuf[:4] == '\x00\x00\xFE\xFF': 
                # 00 00 FE FF  UTF-32, big-endian BOM
                self.result = {'encoding': "UTF-32BE", 'confidence': 1.0}
            elif aBuf[:4] == '\xFE\xFF\x00\x00':
                # FE FF 00 00  UCS-4, unusual octet order BOM (3412)
                self.result = {'encoding': "X-ISO-10646-UCS-4-3412", 'confidence': 1.0}
            elif aBuf[:4] == '\x00\x00\xFF\xFE':
                # 00 00 FF FE  UCS-4, unusual octet order BOM (2143)
                self.result = {'encoding': "X-ISO-10646-UCS-4-2143", 'confidence': 1.0}
            elif aBuf[:2] == '\xFF\xFE':
                # FF FE  UTF-16, little endian BOM
                self.result = {'encoding': "UTF-16LE", 'confidence': 1.0}
            elif aBuf[:2] == '\xFE\xFF':
                # FE FF  UTF-16, big endian BOM
                self.result = {'encoding': "UTF-16BE", 'confidence': 1.0}

从代码中,其实好像也看不出太多问题。

3. 而网上找了些相关的错误的解释,很多说是,由于使用chardet之前,需要先用yourStr.decode("utf16-be")之类的先去解码,然后就可以解决问题了。

而我此处,懒得去详细折腾了。毕竟出现的警告是由于chardet库从1.0.1升级到1.1所导致的。

【总结】

而此处,或许是需要我调用chardet前,也需要处理编码/解码等事情,或许是chardet库本身写的不够好。

但是我的此处的需求是,chardet够用,也就好了,暂时没兴趣去详细调试,等以后有需要时再说。

所以此处的解决办法是:

暂时不把chardet升级到1.1了,还是使用旧的1.0.1的chardet就够用了,也不会有此警告出现。

转载请注明:在路上 » 【已解决】Python脚本运行出错:libs/thirdparty\chardet\universaldetector.py:90: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode – interpreting them as being unequal

发表我的评论
取消评论

表情

Hi,您需要填写昵称和邮箱!

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址
68 queries in 0.108 seconds, using 18.81MB memory