折腾：
【已解决】用Python爬虫框架PySpider实现爬虫爬取百度热榜内容列表
期间，先去给Mac中的Python3安装PySpider
参考
安装：
pip install pyspider
详细日志：
xxx@xxx ~/dev/crifan/python/demo_spider pip install pyspider Collecting pyspider Downloading pyspider-0.3.10.tar.gz (110 kB) |████████████████████████████████| 110 kB 1.1 MB/s Requirement already satisfied: Flask>=0.10 in /Users/xxx/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages (from pyspider) (1.1.1) Requirement already satisfied: Jinja2>=2.7 in /Users/xxx/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages (from pyspider) (2.11.1) Requirement already satisfied: chardet>=2.2 in /Users/xxx/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages (from pyspider) (3.0.4) Requirement already satisfied: cssselect>=0.9 in /Users/xxx/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages (from pyspider) (1.1.0) Requirement already satisfied: lxml in /Users/xxx/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages (from pyspider) (4.5.0) Collecting pycurl Downloading pycurl-7.43.0.5.tar.gz (216 kB) |████████████████████████████████| 216 kB 3.5 MB/s Requirement already satisfied: requests>=2.2 in /Users/xxx/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages (from pyspider) (2.22.0) Collecting Flask-Login>=0.2.11 Downloading Flask_Login-0.5.0-py2.py3-none-any.whl (16 kB) Collecting u-msgpack-python>=1.6 Downloading u_msgpack_python-2.6.0-py2.py3-none-any.whl (9.8 kB) Requirement already satisfied: click>=3.3 in /Users/xxx/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages (from pyspider) (7.0) Requirement already satisfied: six>=1.5.0 in /Users/xxx/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages (from pyspider) (1.14.0) Collecting tblib>=1.3.0 Downloading tblib-1.7.0-py2.py3-none-any.whl (12 kB) Collecting wsgidav>=2.0.0 Downloading WsgiDAV-3.0.3-py2.py3-none-any.whl (170 kB) |████████████████████████████████| 170 kB 9.9 MB/s Collecting tornado<=4.5.3,>=3.2 Downloading tornado-4.5.3.tar.gz (484 kB) |████████████████████████████████| 484 kB 10.5 MB/s Collecting pyquery Downloading pyquery-1.4.1-py2.py3-none-any.whl (22 kB) Requirement already satisfied: itsdangerous>=0.24 in /Users/xxx/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages (from Flask>=0.10->pyspider) (1.1.0) Requirement already satisfied: Werkzeug>=0.15 in /Users/xxx/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages (from Flask>=0.10->pyspider) (1.0.0) Requirement already satisfied: MarkupSafe>=0.23 in /Users/xxx/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages (from Jinja2>=2.7->pyspider) (1.1.1) Requirement already satisfied: certifi>=2017.4.17 in /Users/xxx/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages (from requests>=2.2->pyspider) (2019.11.28) Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /Users/xxx/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages (from requests>=2.2->pyspider) (1.25.8) Requirement already satisfied: idna<2.9,>=2.5 in /Users/xxx/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages (from requests>=2.2->pyspider) (2.8) Collecting defusedxml Downloading defusedxml-0.6.0-py2.py3-none-any.whl (23 kB) Collecting json5 Downloading json5-0.9.5-py2.py3-none-any.whl (17 kB) Requirement already satisfied: PyYAML in /Users/xxx/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages (from wsgidav>=2.0.0->pyspider) (5.3) Using legacy setup.py install for pyspider, since package 'wheel' is not installed. Using legacy setup.py install for pycurl, since package 'wheel' is not installed. Using legacy setup.py install for tornado, since package 'wheel' is not installed. Installing collected packages: pycurl, Flask-Login, u-msgpack-python, tblib, defusedxml, json5, wsgidav, tornado, pyquery, pyspider Running setup.py install for pycurl ... done Attempting uninstall: tornado Found existing installation: tornado 6.0.3 Uninstalling tornado-6.0.3: Successfully uninstalled tornado-6.0.3 Running setup.py install for tornado ... done Running setup.py install for pyspider ... done Successfully installed Flask-Login-0.5.0 defusedxml-0.6.0 json5-0.9.5 pycurl-7.43.0.5 pyquery-1.4.1 pyspider-0.3.10 tblib-1.7.0 tornado-4.5.3 u-msgpack-python-2.6.0 wsgidav-3.0.3
顺带：查看一下安装后的版本和信息
xxx@xxx ~/dev/crifan/python/demo_spider pip show pyspider Name: pyspider Version: 0.3.10 Summary: A Powerful Spider System in Python Home-page: https://github.com/binux/pyspider Author: Roy Binux Author-email: roy@binux.me License: Apache License, Version 2.0 Location: /Users/xxx/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages Requires: Flask, Jinja2, chardet, cssselect, lxml, pycurl, requests, Flask-Login, u-msgpack-python, click, six, tblib, wsgidav, tornado, pyquery Required-by:
启动：
pyspider
此处又遇到之前遇到的问题：
pyspider
Traceback (most recent call last):
File "/Users/xxx/.pyenv/versions/3.8.0/bin/pyspider", line 11, in <module>
load_entry_point('pyspider==0.3.10', 'console_scripts', 'pyspider')()
File "/Users/xxx/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages/pkg_resources/__init__.py", line 489, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/Users/xxx/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2852, in load_entry_point
return ep.load()
File "/Users/xxx/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2443, in load
return self.resolve()
File "/Users/xxx/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2449, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "/Users/xxx/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages/pyspider/run.py", line 231
async=True, get_object=False, no_input=False):
^
SyntaxError: invalid syntax参考自己之前的：
【已解决】mac中pipenv运行pyspider出错：pyspider/run.py async=True SyntaxError: invalid syntax
先去换回Python 3.6吧
以及最好顺带创建一个Python 3.6的虚拟环境，在里面用PySpider
去参考
【已解决】Mac中pyenv安装和设置python 3.7为系统python
重新用pyenv安装一个Python 3.6的：
pyenv install —list 。。。 3.5.8rc1 3.6.0 3.6-dev 3.6.1 3.6.2 3.6.3 3.6.4 3.6.5 3.6.6 3.6.7 3.6.8 3.6.9 3.7.0
先去安装一个3.6的最后一个，版本号是偶数的3.6.8：
pyenv install 3.6.8 python-build: use openssl@1.1 from homebrew python-build: use readline from homebrew Downloading Python-3.6.8.tar.xz... -> https://www.python.org/ftp/python/3.6.8/Python-3.6.8.tar.xz Installing Python-3.6.8... python-build: use readline from homebrew python-build: use zlib from xcode sdk Installed Python-3.6.8 to /Users/xxx/.pyenv/versions/3.6.8
确认已安装：
pyenv versions system 3.6.5 3.6.8 * 3.8.0 (set by PYENV_VERSION environment variable)
然后本地设置为3.6.8：
pyenv local 3.6.8 pyenv local 3.6.8
创建虚拟环境，好像也不用了？还是用吧。防止和全局的pyspider干扰？
不过还是给全局的安装PySpider也是可以的
所以不用虚拟环境了。直接去安装：
pip install pyspider Requirement already satisfied: pyspider in /Users/xxx/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages (0.3.10) Requirement already satisfied: Flask>=0.10 in /Users/xxx/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages (from pyspider) (1.1.1) Requirement already satisfied: Jinja2>=2.7 in /Users/xxx/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages (from pyspider) (2.11.1) Requirement already satisfied: chardet>=2.2 in /Users/xxx/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages (from pyspider) (3.0.4) Requirement already satisfied: cssselect>=0.9 in /Users/xxx/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages (from pyspider) (1.1.0) Requirement already satisfied: lxml in /Users/xxx/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages (from pyspider) (4.5.0) Requirement already satisfied: pycurl in /Users/xxx/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages (from pyspider) (7.43.0.5) Requirement already satisfied: requests>=2.2 in /Users/xxx/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages (from pyspider) (2.22.0) Requirement already satisfied: Flask-Login>=0.2.11 in /Users/xxx/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages (from pyspider) (0.5.0) Requirement already satisfied: u-msgpack-python>=1.6 in /Users/xxx/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages (from pyspider) (2.6.0) Requirement already satisfied: click>=3.3 in /Users/xxx/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages (from pyspider) (7.0) Requirement already satisfied: six>=1.5.0 in /Users/xxx/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages (from pyspider) (1.14.0) Requirement already satisfied: tblib>=1.3.0 in /Users/xxx/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages (from pyspider) (1.7.0) Requirement already satisfied: wsgidav>=2.0.0 in /Users/xxx/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages (from pyspider) (3.0.3) Requirement already satisfied: tornado<=4.5.3,>=3.2 in /Users/xxx/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages (from pyspider) (4.5.3) Requirement already satisfied: pyquery in /Users/xxx/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages (from pyspider) (1.4.1) Requirement already satisfied: Werkzeug>=0.15 in /Users/xxx/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages (from Flask>=0.10->pyspider) (1.0.0) Requirement already satisfied: itsdangerous>=0.24 in /Users/xxx/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages (from Flask>=0.10->pyspider) (1.1.0) Requirement already satisfied: MarkupSafe>=0.23 in /Users/xxx/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages (from Jinja2>=2.7->pyspider) (1.1.1) Requirement already satisfied: certifi>=2017.4.17 in /Users/xxx/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages (from requests>=2.2->pyspider) (2019.11.28) Requirement already satisfied: idna<2.9,>=2.5 in /Users/xxx/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages (from requests>=2.2->pyspider) (2.8) Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /Users/xxx/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages (from requests>=2.2->pyspider) (1.25.8) Requirement already satisfied: PyYAML in /Users/xxx/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages (from wsgidav>=2.0.0->pyspider) (5.3) Requirement already satisfied: defusedxml in /Users/xxx/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages (from wsgidav>=2.0.0->pyspider) (0.6.0) Requirement already satisfied: json5 in /Users/xxx/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages (from wsgidav>=2.0.0->pyspider) (0.9.5)
已安装了。
就是用的全局的PySpider。
然后去运行，看看是否能正常运行
错误依旧。
发现估计是此处的问题：
VSCode中左下角的Python还是旧的3.8的：
重启VSCode：
好像问题依旧。
重新切换一下：
然后重启终端：
pyenv shell 3.6.8 xxx@xxx ~/dev/crifan/python/demo_spider pyenv shell 3.6.8 xxx@xxx ~/dev/crifan/python/demo_spider pyenv local 3.6.8
是我们要的 3.6.8的了。
然后再去给3.6.8的Python环境，安装PySpider：
xxx@xxx ~/dev/crifan/python/demo_spider pip install pyspider Collecting pyspider Using cached https://files.pythonhosted.org/packages/d0/97/d6062c928f53d899ff2a8538fed11d4d425ba3d27c96248a2c601c1c9fef/pyspider-0.3.10.tar.gz Collecting Flask>=0.10 (from pyspider) Downloading https://files.pythonhosted.org/packages/f2/28/2a03252dfb9ebf377f40fba6a7841b47083260bf8bd8e737b0c6952df83f/Flask-1.1.2-py2.py3-none-any.whl (94kB) 100% |████████████████████████████████| 102kB 622kB/s Collecting Jinja2>=2.7 (from pyspider) Downloading https://files.pythonhosted.org/packages/30/9e/f663a2aa66a09d838042ae1a2c5659828bb9b41ea3a6efa20a20fd92b121/Jinja2-2.11.2-py2.py3-none-any.whl (125kB) 100% |████████████████████████████████| 133kB 1.7MB/s Collecting chardet>=2.2 (from pyspider) Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl Collecting cssselect>=0.9 (from pyspider) Using cached https://files.pythonhosted.org/packages/3b/d4/3b5c17f00cce85b9a1e6f91096e1cc8e8ede2e1be8e96b87ce1ed09e92c5/cssselect-1.1.0-py2.py3-none-any.whl Collecting lxml (from pyspider) Downloading https://files.pythonhosted.org/packages/ad/66/8e3139f8bd3200777c43b508a71cac92c17d96c58fb0117a2ccce3fc3f4a/lxml-4.5.2-cp36-cp36m-macosx_10_9_x86_64.whl (4.5MB) 100% |████████████████████████████████| 4.5MB 1.4MB/s Collecting pycurl (from pyspider) Using cached https://files.pythonhosted.org/packages/ef/05/4b773f74f830a90a326b06f9b24e65506302ab049e825a3c0b60b1a6e26a/pycurl-7.43.0.5.tar.gz Collecting requests>=2.2 (from pyspider) Downloading https://files.pythonhosted.org/packages/45/1e/0c169c6a5381e241ba7404532c16a21d86ab872c9bed8bdcd4c423954103/requests-2.24.0-py2.py3-none-any.whl (61kB) 100% |████████████████████████████████| 71kB 9.5MB/s Collecting Flask-Login>=0.2.11 (from pyspider) Using cached https://files.pythonhosted.org/packages/2b/83/ac5bf3279f969704fc1e63f050c50e10985e50fd340e6069ec7e09df5442/Flask_Login-0.5.0-py2.py3-none-any.whl Collecting u-msgpack-python>=1.6 (from pyspider) Using cached https://files.pythonhosted.org/packages/58/96/24a68e6aad8b47acf8d0ca898138799cd0f901e2f8e5f6095692c984a45c/u_msgpack_python-2.6.0-py2.py3-none-any.whl Collecting click>=3.3 (from pyspider) Downloading https://files.pythonhosted.org/packages/d2/3d/fa76db83bf75c4f8d338c2fd15c8d33fdd7ad23a9b5e57eb6c5de26b430e/click-7.1.2-py2.py3-none-any.whl (82kB) 100% |████████████████████████████████| 92kB 12.5MB/s Collecting six>=1.5.0 (from pyspider) Downloading https://files.pythonhosted.org/packages/ee/ff/48bde5c0f013094d729fe4b0316ba2a24774b3ff1c52d924a8a4cb04078a/six-1.15.0-py2.py3-none-any.whl Collecting tblib>=1.3.0 (from pyspider) Using cached https://files.pythonhosted.org/packages/f8/cd/2fad4add11c8837e72f50a30e2bda30e67a10d70462f826b291443a55c7d/tblib-1.7.0-py2.py3-none-any.whl Collecting wsgidav>=2.0.0 (from pyspider) Using cached https://files.pythonhosted.org/packages/58/10/87025d42662f06455b7a568ccdd51c5a40e6d9642ee202bd28776d08883a/WsgiDAV-3.0.3-py2.py3-none-any.whl Collecting tornado<=4.5.3,>=3.2 (from pyspider) Using cached https://files.pythonhosted.org/packages/e3/7b/e29ab3d51c8df66922fea216e2bddfcb6430fb29620e5165b16a216e0d3c/tornado-4.5.3.tar.gz Collecting pyquery (from pyspider) Using cached https://files.pythonhosted.org/packages/78/43/95d42e386c61cb639d1a0b94f0c0b9f0b7d6b981ad3c043a836c8b5bc68b/pyquery-1.4.1-py2.py3-none-any.whl Collecting itsdangerous>=0.24 (from Flask>=0.10->pyspider) Using cached https://files.pythonhosted.org/packages/76/ae/44b03b253d6fade317f32c24d100b3b35c2239807046a4c953c7b89fa49e/itsdangerous-1.1.0-py2.py3-none-any.whl Collecting Werkzeug>=0.15 (from Flask>=0.10->pyspider) Downloading https://files.pythonhosted.org/packages/cc/94/5f7079a0e00bd6863ef8f1da638721e9da21e5bacee597595b318f71d62e/Werkzeug-1.0.1-py2.py3-none-any.whl (298kB) 100% |████████████████████████████████| 307kB 1.7MB/s Collecting MarkupSafe>=0.23 (from Jinja2>=2.7->pyspider) Using cached https://files.pythonhosted.org/packages/f0/00/a6aea33f5598b080b86d6b6d1214b51afe3ffa6100b902d5aa465080083f/MarkupSafe-1.1.1-cp36-cp36m-macosx_10_6_intel.whl Collecting certifi>=2017.4.17 (from requests>=2.2->pyspider) Downloading https://files.pythonhosted.org/packages/5e/c4/6c4fe722df5343c33226f0b4e0bb042e4dc13483228b4718baf286f86d87/certifi-2020.6.20-py2.py3-none-any.whl (156kB) 100% |████████████████████████████████| 163kB 1.6MB/s Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests>=2.2->pyspider) Downloading https://files.pythonhosted.org/packages/9f/f0/a391d1463ebb1b233795cabfc0ef38d3db4442339de68f847026199e69d7/urllib3-1.25.10-py2.py3-none-any.whl (127kB) 100% |████████████████████████████████| 133kB 6.7MB/s Collecting idna<3,>=2.5 (from requests>=2.2->pyspider) Downloading https://files.pythonhosted.org/packages/a2/38/928ddce2273eaa564f6f50de919327bf3a00f091b5baba8dfa9460f3a8a8/idna-2.10-py2.py3-none-any.whl (58kB) 100% |████████████████████████████████| 61kB 9.5MB/s Collecting defusedxml (from wsgidav>=2.0.0->pyspider) Using cached https://files.pythonhosted.org/packages/06/74/9b387472866358ebc08732de3da6dc48e44b0aacd2ddaa5cb85ab7e986a2/defusedxml-0.6.0-py2.py3-none-any.whl Collecting PyYAML (from wsgidav>=2.0.0->pyspider) Downloading https://files.pythonhosted.org/packages/64/c2/b80047c7ac2478f9501676c988a5411ed5572f35d1beff9cae07d321512c/PyYAML-5.3.1.tar.gz (269kB) 100% |████████████████████████████████| 276kB 2.9MB/s Collecting json5 (from wsgidav>=2.0.0->pyspider) Using cached https://files.pythonhosted.org/packages/2b/81/22bf51a5bc60dde18bb6164fd597f18ee683de8670e141364d9c432dd3cf/json5-0.9.5-py2.py3-none-any.whl Installing collected packages: MarkupSafe, Jinja2, itsdangerous, Werkzeug, click, Flask, chardet, cssselect, lxml, pycurl, certifi, urllib3, idna, requests, Flask-Login, u-msgpack-python, six, tblib, defusedxml, PyYAML, json5, wsgidav, tornado, pyquery, pyspider Running setup.py install for pycurl ... done Running setup.py install for PyYAML ... done Running setup.py install for tornado ... done Running setup.py install for pyspider ... done Successfully installed Flask-1.1.2 Flask-Login-0.5.0 Jinja2-2.11.2 MarkupSafe-1.1.1 PyYAML-5.3.1 Werkzeug-1.0.1 certifi-2020.6.20 chardet-3.0.4 click-7.1.2 cssselect-1.1.0 defusedxml-0.6.0 idna-2.10 itsdangerous-1.1.0 json5-0.9.5 lxml-4.5.2 pycurl-7.43.0.5 pyquery-1.4.1 pyspider-0.3.10 requests-2.24.0 six-1.15.0 tblib-1.7.0 tornado-4.5.3 u-msgpack-python-2.6.0 urllib3-1.25.10 wsgidav-3.0.3
【总结】
此处Mac中给Python 3.8安装PySpider，结果后来无法启动，由于PySpider版本没更新，不支持超过3.6的Python
解决办法是
暂时换成Python 3.6，才可以。
此处用pyenv安装了Python 3.6.8的环境：
pyenv install 3.6.8
再把本地环境设置为3.6.8
pyenv local 3.6.8
重启VSCode和终端，即可确保已处于Python 3.6.8。
即可正常运行PySpider。
转载请注明：在路上 » 【已解决】Mac中给Python3安装PySpider