最新消息:20210917 已从crifan.com换到crifan.org

【已解决】pyspider运行出错:ImportError pycurl libcurl link-time ssl backend (openssl) is different from compile-time ssl backend (none/other)

pyspider crifan 3005浏览 0评论

折腾:

【已解决】写Python爬虫爬取汽车之家品牌车系车型数据

期间,在pipenv中安装好了pyspider后,去运行:

pyspider

结果出错:

<code>➜  AutocarData pyspider
phantomjs fetcher running on port 25555
[I 180421 14:40:28 result_worker:49] result_worker starting...
Process Process-4:
Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/pyspider/run.py", line 236, in fetcher
    Fetcher = load_cls(None, None, fetcher_cls)
  File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/pyspider/run.py", line 48, in load_cls
    return utils.load_object(value)
  File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/pyspider/libs/utils.py", line 369, in load_object
    module = __import__(module_name, globals(), locals(), [object_name])
  File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/pyspider/fetcher/__init__.py", line 1, in &lt;module&gt;
    from .tornado_fetcher import Fetcher
  File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/pyspider/fetcher/tornado_fetcher.py", line 30, in &lt;module&gt;
    from tornado.curl_httpclient import CurlAsyncHTTPClient
  File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/tornado/curl_httpclient.py", line 24, in &lt;module&gt;
    import pycurl  # type: ignore
ImportError: pycurl: libcurl link-time ssl backend (openssl) is different from compile-time ssl backend (none/other)
[I 180421 14:40:29 processor:211] processor starting...
Traceback (most recent call last):
  File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/bin/pyspider", line 11, in &lt;module&gt;
    sys.exit(main())
  File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/pyspider/run.py", line 754, in main
    cli()
  File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/click/core.py", line 1043, in invoke
    return Command.invoke(self, ctx)
  File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/pyspider/run.py", line 165, in cli
    ctx.invoke(all)
  File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/pyspider/run.py", line 497, in all
    ctx.invoke(webui, **webui_config)
  File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/pyspider/run.py", line 333, in webui
    app = load_cls(None, None, webui_instance)
  File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/pyspider/run.py", line 48, in load_cls
    return utils.load_object(value)
  File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/pyspider/libs/utils.py", line 369, in load_object
    module = __import__(module_name, globals(), locals(), [object_name])
  File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/pyspider/webui/__init__.py", line 8, in &lt;module&gt;
    from . import app, index, debug, task, result, login
  File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/pyspider/webui/app.py", line 17, in &lt;module&gt;
    from pyspider.fetcher import tornado_fetcher
  File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/pyspider/fetcher/__init__.py", line 1, in &lt;module&gt;
    from .tornado_fetcher import Fetcher
  File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/pyspider/fetcher/tornado_fetcher.py", line 30, in &lt;module&gt;
    from tornado.curl_httpclient import CurlAsyncHTTPClient
  File "/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/tornado/curl_httpclient.py", line 24, in &lt;module&gt;
    import pycurl  # type: ignore
ImportError: pycurl: libcurl link-time ssl backend (openssl) is different from compile-time ssl backend (none/other)
➜  AutocarData
</code>

搜:

pyspider ImportError pycurl libcurl link-time ssl backend (openssl) is different from compile-time ssl backend (none/other)

解决pycurl安装错误 | 开心&努力

ImportError: pycurl: libcurl link-time ssl backend (openssl) is different from compile-time ssl backend (none/other) · Issue #4 · transloadit/python-sdk

python – SSL backend error when using OpenSSL – Stack Overflow

macos下安装pyspider出现ImportError: pycurl错误 – CSDN博客

<code>➜  AutocarData pip uninstall pycurl
Uninstalling pycurl-7.43.0.1:
  Would remove:
    /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/curl/*
    /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/pycurl-7.43.0.1.dist-info/*
    /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/lib/python3.6/site-packages/pycurl.cpython-36m-darwin.so
    /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/AUTHORS
    /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/COPYING-LGPL
    /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/COPYING-MIT
    /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/ChangeLog
    /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/INSTALL.rst
    /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/README.rst
    /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/RELEASE-NOTES.rst
    /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/examples/basicfirst.py
    /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/examples/file_upload.py
    /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/examples/linksys.py
    /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/examples/quickstart/file_upload_buffer.py
    /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/examples/quickstart/file_upload_real.py
    /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/examples/quickstart/file_upload_real_fancy.py
    /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/examples/quickstart/follow_redirect.py
    /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/examples/quickstart/form_post.py
    /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/examples/quickstart/get.py
    /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/examples/quickstart/get_python2.py
    /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/examples/quickstart/get_python3.py
    /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/examples/quickstart/response_headers.py
    /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/examples/quickstart/response_info.py
    /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/examples/quickstart/write_file.py
    /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/examples/retriever-multi.py
    /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/examples/retriever.py
    /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/examples/sfquery.py
    /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/examples/smtp.py
    /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/examples/ssh_keyfunction.py
    /Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/share/doc/pycurl/examples/xmlrpc_curl.py
Proceed (y/n)? y
  Successfully uninstalled pycurl-7.43.0.1
➜  AutocarData export PYCURL_SSL_LIBRARY=openssl
➜  AutocarData pip install pycrul
Collecting pycrul
  Could not find a version that satisfies the requirement pycrul (from versions: )
No matching distribution found for pycrul
➜  AutocarData where python
/Users/crifan/.local/share/virtualenvs/AutocarData-xI-iqIq4/bin/python
/usr/bin/python
➜  AutocarData python --version
Python 3.6.4
</code>

还是不行。

注意到:

估计是由于此处的Python3的环境,导致出问题的?

加上路径,就可以了:

<code>➜  AutocarData export LDFLAGS=-L/usr/local/opt/openssl/lib;export CPPFLAGS=-I/usr/local/opt/openssl/include;pip install pycurl --compile --no-cache-dir

Collecting pycurl
  Downloading https://files.pythonhosted.org/packages/77/d9/d272b38e6e25d2686e22f6058820298dadead69340b1c57ff84c87ef81f0/pycurl-7.43.0.1.tar.gz (195kB)
    100% |████████████████████████████████| 204kB 292kB/s
Installing collected packages: pycurl
  Running setup.py install for pycurl ... done
Successfully installed pycurl-7.43.0.1
</code>

然后再去运行pyspider看看:

结果(虽然出现了其他错误,但是)可以运行了。

【总结】

此处解决办法:

<code>pip uninstall pycurl
export PYCURL_SSL_LIBRARY=openssl
export LDFLAGS=-L/usr/local/opt/openssl/lib;export CPPFLAGS=-I/usr/local/opt/openssl/include;pip install pycurl --compile --no-cache-dir
</code>

即可。

转载请注明:在路上 » 【已解决】pyspider运行出错:ImportError pycurl libcurl link-time ssl backend (openssl) is different from compile-time ssl backend (none/other)

发表我的评论
取消评论

表情

Hi,您需要填写昵称和邮箱!

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址
95 queries in 0.185 seconds, using 23.48MB memory