pictuga
/
morss
1
1
Fork 0
Code Issues Pull Requests Actions Packages Releases Activity

Only use chardet on 2k letters 

Takes forever otherwise
master
pictuga 2017-07-16 23:59:06 +02:00
parent 6f0efd5802
commit df22396838
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
morss/crawler.py
View File

@ -98,7 +98,7 @@ def detect_encoding(data, con=None):
if match: if match:
return match.groups()[0].lower().decode() return match.groups()[0].lower().decode()
enc = chardet.detect(data)['encoding'] enc = chardet.detect(data[-2000:])['encoding']
if enc and enc != 'ascii': if enc and enc != 'ascii':
return enc return enc