折腾:
期间,在pipenv中安装好了pyspider后,去运行:
pyspider
结果出错:
<code>➜ AutocarData pyspider phantomjs fetcher running on port 25555 [I 180421 14:40:28 result_worker:49] result_worker starting... Process Process-4: Traceback (most recent call last): File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/process.py", line 93, in run self._target(*self._args, **self._kwargs) File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/click/core.py", line 535, in invoke return callback(*args, **kwargs) File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func return f(get_current_context(), *args, **kwargs) File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/pyspider/run.py", line 236, in fetcher Fetcher = load_cls(None, None, fetcher_cls) File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/pyspider/run.py", line 48, in load_cls return utils.load_object(value) File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/pyspider/libs/utils.py", line 369, in load_object module = __import__(module_name, globals(), locals(), [object_name]) File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/pyspider/fetcher/__init__.py", line 1, in <module> from .tornado_fetcher import Fetcher File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/pyspider/fetcher/tornado_fetcher.py", line 30, in <module> from tornado.curl_httpclient import CurlAsyncHTTPClient File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/tornado/curl_httpclient.py", line 24, in <module> import pycurl # type: ignore ImportError: pycurl: libcurl link-time ssl backend (openssl) is different from compile-time ssl backend (none/other) [I 180421 14:40:29 processor:211] processor starting... Traceback (most recent call last): File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/bin/pyspider", line 11, in <module> sys.exit(main()) File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/pyspider/run.py", line 754, in main cli() File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/click/core.py", line 722, in __call__ return self.main(*args, **kwargs) File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/click/core.py", line 697, in main rv = self.invoke(ctx) File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/click/core.py", line 1043, in invoke return Command.invoke(self, ctx) File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/click/core.py", line 895, in invoke return ctx.invoke(self.callback, **ctx.params) File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/click/core.py", line 535, in invoke return callback(*args, **kwargs) File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func return f(get_current_context(), *args, **kwargs) File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/pyspider/run.py", line 165, in cli ctx.invoke(all) File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/click/core.py", line 535, in invoke return callback(*args, **kwargs) File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func return f(get_current_context(), *args, **kwargs) File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/pyspider/run.py", line 497, in all ctx.invoke(webui, **webui_config) File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/click/core.py", line 535, in invoke return callback(*args, **kwargs) File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func return f(get_current_context(), *args, **kwargs) File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/pyspider/run.py", line 333, in webui app = load_cls(None, None, webui_instance) File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/pyspider/run.py", line 48, in load_cls return utils.load_object(value) File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/pyspider/libs/utils.py", line 369, in load_object module = __import__(module_name, globals(), locals(), [object_name]) File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/pyspider/webui/__init__.py", line 8, in <module> from . import app, index, debug, task, result, login File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/pyspider/webui/app.py", line 17, in <module> from pyspider.fetcher import tornado_fetcher File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/pyspider/fetcher/__init__.py", line 1, in <module> from .tornado_fetcher import Fetcher File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/pyspider/fetcher/tornado_fetcher.py", line 30, in <module> from tornado.curl_httpclient import CurlAsyncHTTPClient File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/tornado/curl_httpclient.py", line 24, in <module> import pycurl # type: ignore ImportError: pycurl: libcurl link-time ssl backend (openssl) is different from compile-time ssl backend (none/other) ➜ AutocarData </code>
搜:
pyspider ImportError pycurl libcurl link-time ssl backend (openssl) is different from compile-time ssl backend (none/other)
python – SSL backend error when using OpenSSL – Stack Overflow
macos下安装pyspider出现ImportError: pycurl错误 – CSDN博客
<code>➜ AutocarData pip uninstall pycurl Uninstalling pycurl-7.43.0.1: Would remove: /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/curl/* /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/pycurl-7.43.0.1.dist-info/* /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/pycurl.cpython-36m-darwin.so /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/AUTHORS /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/COPYING-LGPL /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/COPYING-MIT /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/ChangeLog /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/INSTALL.rst /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/README.rst /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/RELEASE-NOTES.rst /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/examples/basicfirst.py /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/examples/file_upload.py /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/examples/linksys.py /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/examples/quickstart/file_upload_buffer.py /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/examples/quickstart/file_upload_real.py /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/examples/quickstart/file_upload_real_fancy.py /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/examples/quickstart/follow_redirect.py /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/examples/quickstart/form_post.py /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/examples/quickstart/get.py /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/examples/quickstart/get_python2.py /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/examples/quickstart/get_python3.py /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/examples/quickstart/response_headers.py /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/examples/quickstart/response_info.py /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/examples/quickstart/write_file.py /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/examples/retriever-multi.py /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/examples/retriever.py /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/examples/sfquery.py /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/examples/smtp.py /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/examples/ssh_keyfunction.py /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/examples/xmlrpc_curl.py Proceed (y/n)? y Successfully uninstalled pycurl-7.43.0.1 ➜ AutocarData export PYCURL_SSL_LIBRARY=openssl ➜ AutocarData pip install pycrul Collecting pycrul Could not find a version that satisfies the requirement pycrul (from versions: ) No matching distribution found for pycrul ➜ AutocarData where python /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/bin/python /usr/bin/python ➜ AutocarData python --version Python 3.6.4 </code>
还是不行。
注意到:
估计是由于此处的Python3的环境,导致出问题的?
加上路径,就可以了:
<code>➜ AutocarData export LDFLAGS=-L/usr/local/opt/openssl/lib;export CPPFLAGS=-I/usr/local/opt/openssl/include;pip install pycurl --compile --no-cache-dir Collecting pycurl Downloading https://files.pythonhosted.org/packages/77/d9/d272b38e6e25d2686e22f6058820298dadead69340b1c57ff84c87ef81f0/pycurl-7.43.0.1.tar.gz (195kB) 100% |████████████████████████████████| 204kB 292kB/s Installing collected packages: pycurl Running setup.py install for pycurl ... done Successfully installed pycurl-7.43.0.1 </code>
然后再去运行pyspider看看:
结果(虽然出现了其他错误,但是)可以运行了。
【总结】
此处解决办法:
<code>pip uninstall pycurl export PYCURL_SSL_LIBRARY=openssl export LDFLAGS=-L/usr/local/opt/openssl/lib;export CPPFLAGS=-I/usr/local/opt/openssl/include;pip install pycurl --compile --no-cache-dir </code>
即可。
转载请注明:在路上 » 【已解决】pyspider运行出错:ImportError pycurl libcurl link-time ssl backend (openssl) is different from compile-time ssl backend (none/other)