最新消息:20190717 VPS服务器:Vultr新加坡,WordPress主题:大前端D8,统一介绍入口:关于

【已解决】Mac中恢复或重新搭建PySpider开发环境

Mac crifan 191浏览 0评论
折腾:
【未解决】用Python爬取汽车之家的车型车系详细数据
期间,先去恢复PySpider环境。
现在先去参考之前已有代码去
恢复环境
pyspider_src/Pipfile
[[source]]
#url = "https://pypi.python.org/simple"
url = "https://pypi.tuna.tsinghua.edu.cn/simple"
verify_ssl = true
name = "pypi"


[packages]
pymysql = "*"
mysql = "*"
mysqlclient = "*"
mysql-connector-python = "*"


[dev-packages]


[requires]
python_version = "3.6"
然后去:
 pipenv install
卡死了。换:
pipenv install --skip-lock
继续:
pipenv shell
去看看:
pipenv graph
mysql==0.0.2
  - mysqlclient [required: Any, installed: 2.0.1]
mysql-connector-python==8.0.21
  - protobuf [required: >=3.0.0, installed: 3.12.4]
    - setuptools [required: Any, installed: 45.3.0]
    - six [required: >=1.9, installed: 1.15.0]
PyMySQL==0.10.0
去安装:
pipenv install --skip-lock pyspider
即可。
然后去看看能否运行pyspider
pyspider
但是没法启动。
重新安装:
pipenv install
但是由于自己的Mac本身有问题:
VSCode的终端中pyspider输入了都没反应
-》后来发现本身Mac系统有问题
无法正常打开各种app,包括网易有道词典
以及VSCode中Python插件都无法正常加载
所以放弃,换了公司Mac。
然后安装全局PySpider
  pyenv version
3.6.5 (set by /Users/xxx/.python-version)
 [email protected]  ~  pip install pyspider
运行
pyspider
报错:
ImportError: pycurl: libcurl link-time ssl backend (openssl) is different from compile-time ssl backend (none/other)
参考之前的去:
pip uninstall pycurl
export PYCURL_SSL_LIBRARY=openssl
export LDFLAGS=-L/usr/local/opt/openssl/lib;export CPPFLAGS=-I/usr/local/opt/openssl/include;pip install pycurl --compile --no-cache-dir
然后又报错:
  - Deprecated option 'domaincontroller': use 'http_authenticator.domain_controller' instead.
Error: Could not create web server listening on port 25555
参考之前的:
【已解决】Mac中启动PySpider
去杀掉进程重启
不过问题依旧,去:
pip install wsgidav==2.4.1
然后又:
ImportError: cannot import name 'DispatcherMiddleware'
继续:
pip install werkzeug==0.16.1
终于可以了:
 pyspider           
phantomjs fetcher running on port 25555
[I 200814 20:31:01 result_worker:49] result_worker starting...
[I 200814 20:31:01 processor:211] processor starting...
[I 200814 20:31:01 tornado_fetcher:638] fetcher starting...
[I 200814 20:31:01 scheduler:647] scheduler starting...
[I 200814 20:31:01 scheduler:782] scheduler.xmlrpc listening on 127.0.0.1:23333
[I 200814 20:31:01 scheduler:586] in 5m: new:0,success:0,retry:0,failed:0
[I 200814 20:31:01 app:76] webui running on 0.0.0.0:5000

转载请注明:在路上 » 【已解决】Mac中恢复或重新搭建PySpider开发环境

发表我的评论
取消评论

表情

Hi,您需要填写昵称和邮箱!

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址
86 queries in 0.107 seconds, using 20.77MB memory