折腾：
【未解决】Win中VMWare中macOS中调试抓包项目
期间，调试
【未解决】Win中尝试访问VMWare中macOS中8100端口的wda服务
期间发现：
iOS中app无法访问网络了：
网络异常，请检查网络设置
请检查您当前的网络环境，如果其他App可以正常使用，请到设置-》斑马AI可-无线数据中允许斑马AI课访问网络。检查后，点击重试按钮。
去看了看，果然是网络问题：
当前WiFi网络加了代理：
但是当前忘了开启代理了。
所以去开启mitmdump代理
crifanli@crifanlideMac ~/dev/DevRoot/appcrawler master ● pwd /Users/crifanli/dev/DevRoot/appcrawler crifanli@crifanlideMac ~/dev/DevRoot/appcrawler master ● ll total 120 -rw-r--r-- 1 crifanli staff 27K 6 23 16:20 README.md -rw-r--r-- 1 crifanli staff 2.4K 6 23 09:14 analyzeStart.py -rw-r--r-- 1 crifanli staff 6.2K 6 23 16:20 crawlerStart.py drwxr-xr-x 3 crifanli staff 96B 6 28 10:48 data drwxr-xr-x 2 crifanli staff 64B 6 28 16:29 debug drwxr-xr-x 7 crifanli staff 224B 6 23 09:14 gameAutomation drwxr-xr-x 5 crifanli staff 160B 6 23 09:14 iOSAutomation drwxr-xr-x 3 crifanli staff 96B 6 23 16:29 logging drwxr-xr-x 24 crifanli staff 768B 6 23 16:20 middleware drwxr-xr-x 7 crifanli staff 224B 6 23 16:20 platformIntegration drwxr-xr-x 13 crifanli staff 416B 6 23 16:20 process -rw-r--r-- 1 crifanli staff 2.6K 6 23 09:14 queryService.py -rw-r--r-- 1 crifanli staff 2.6K 6 23 09:14 queryService1.py -rw-r--r-- 1 crifanli staff 4.5K 6 23 15:32 requirements.txt drwxr-xr-x 7 crifanli staff 224B 6 23 16:26 service drwxr-xr-x 8 crifanli staff 256B 6 24 13:54 src -rw-r--r-- 1 crifanli staff 72B 6 23 09:14 stop_mongod.js drwxr-xr-x 17 crifanli staff 544B 6 24 11:26 task drwxr-xr-x 5 crifanli staff 160B 6 23 09:14 tmp drwxr-xr-x 17 crifanli staff 544B 6 23 09:14 tool drwxr-xr-x 6 crifanli staff 192B 6 23 16:26 utils drwxr-xr-x 6 crifanli staff 192B 6 23 11:46 venv crifanli@crifanlideMac ~/dev/DevRoot/appcrawler master ● mitmdump -k -p 8081 -s middleware/Save1.py zsh: command not found: mitmdump
很明显，macOS系统全局还没安装mitmproxy呢。去安装。
虽然好像直接brew即可，不过也去找之前帖子，参考看看。
【已解决】Mac中安装Mitmdump和启动服务
pip install mitmproxy
而不用brew安装，否则后续会出现yaml找不到的问题
不过也要先去macOS中安装用pip安装yaml，所以应该是
pip instal pyyaml pip install mitmproxy
详细log
✘ crifanli@crifanlideMac ~/dev/DevRoot/appcrawler master ● pip install pyyaml Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple Processing /Users/crifanli/Library/Caches/pip/wheels/48/41/22/8d283b016a1f9f8eac6d9f3c5b99cb58aabf8681225ab4f6dd/PyYAML-5.3.1-cp38-cp38-macosx_10_14_x86_64.whl Installing collected packages: pyyaml Successfully installed pyyaml-5.3.1
以及：
crifanli@crifanlideMac ~/dev/DevRoot/appcrawler master ● pip install mitmproxy Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple Collecting mitmproxy Downloading https://pypi.tuna.tsinghua.edu.cn/packages/d3/b9/fdfa51a2ae6ab968068a1b682e0c9efb76cc2f0493cf3e4fab33166d8345/mitmproxy-5.1.1-py3-none-any.whl (1.3 MB) |████████████████████████████████| 1.3 MB 1.3 MB/s Collecting pyOpenSSL<19.2,>=19.1.0 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/9e/de/f8342b68fa9e981d348039954657bdf681b2ab93de27443be51865ffa310/pyOpenSSL-19.1.0-py2.py3-none-any.whl (53 kB) |████████████████████████████████| 53 kB 5.3 MB/s Collecting wsproto<0.16,>=0.14 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/d5/8b/96575a9a73591ce8e5e519375e30565bf9d299d0ced98a2970c225abedf4/wsproto-0.15.0-py2.py3-none-any.whl (24 kB) Collecting tornado<7,>=4.3 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/95/84/119a46d494f008969bf0c775cb2c6b3579d3c4cc1bb1b41a022aa93ee242/tornado-6.0.4.tar.gz (496 kB) |████████████████████████████████| 496 kB 11.2 MB/s Collecting cryptography<3.0,>=2.9 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/0f/ed/93c2217e732e75c21477c350b2d10a77e08fe2e0065c14644bdb110acfdd/cryptography-2.9.2-cp35-abi3-macosx_10_9_x86_64.whl (1.8 MB) |████████████████████████████████| 1.8 MB 30.7 MB/s Collecting h2<4,>=3.2.0 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/25/de/da019bcc539eeab02f6d45836f23858ac467f584bfec7a526ef200242afe/h2-3.2.0-py2.py3-none-any.whl (65 kB) |████████████████████████████████| 65 kB 5.5 MB/s Requirement already satisfied: click<8,>=7.0 in /Users/crifanli/.pyenv/versions/3.8.3/lib/python3.8/site-packages (from mitmproxy) (7.1.2) Collecting pyasn1<0.5,>=0.3.1 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/62/1e/a94a8d635fa3ce4cfc7f506003548d0a2447ae76fd5ca53932970fe3053f/pyasn1-0.4.8-py2.py3-none-any.whl (77 kB) |████████████████████████████████| 77 kB 142 kB/s Collecting urwid<2.2,>=2.1.0 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/45/dd/d57924f77b0914f8a61c81222647888fbb583f89168a376ffeb5613b02a6/urwid-2.1.0.tar.gz (630 kB) |████████████████████████████████| 630 kB 691 kB/s Collecting zstandard<0.14,>=0.11 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/c6/79/a7b31258dfc071344e5e21976a395310330e72ffad0e3d12a68e3d2c82fe/zstandard-0.13.0-cp38-cp38-macosx_10_9_x86_64.whl (460 kB) |████████████████████████████████| 460 kB 1.1 MB/s Requirement already satisfied: flask<1.2,>=1.1.1 in /Users/crifanli/.pyenv/versions/3.8.3/lib/python3.8/site-packages (from mitmproxy) (1.1.2) Collecting pyparsing<2.5,>=2.4.2 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/8a/bb/488841f56197b13700afd5658fc279a2025a39e22449b7cf29864669b15d/pyparsing-2.4.7-py2.py3-none-any.whl (67 kB) |████████████████████████████████| 67 kB 3.3 MB/s Requirement already satisfied: certifi>=2019.9.11 in /Users/crifanli/.pyenv/versions/3.8.3/lib/python3.8/site-packages (from mitmproxy) (2020.6.20) Processing /Users/crifanli/Library/Caches/pip/wheels/8c/3c/0a/2b4abe85e05a2c31ef23e71e79d7e8c20b1c42dc56b52912c6/kaitaistruct-0.8-py2.py3-none-any.whl Collecting Brotli<1.1,>=1.0 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/16/f8/88122cbd692b56f8bdb9e16b6bb7aad6cda2a6057d28dcaa1a409b6aa0a6/Brotli-1.0.7-cp38-cp38-macosx_10_9_x86_64.whl (412 kB) |████████████████████████████████| 412 kB 257 kB/s Collecting ruamel.yaml<0.17,>=0.16 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/a6/92/59af3e38227b9cc14520bf1e59516d99ceca53e3b8448094248171e9432b/ruamel.yaml-0.16.10-py2.py3-none-any.whl (111 kB) |████████████████████████████████| 111 kB 3.8 MB/s Processing /Users/crifanli/Library/Caches/pip/wheels/22/81/56/0ae6d0abd812c969a7bb0a56e80667d5f0d6aa57d01bd3fe1d/blinker-1.4-py3-none-any.whl Collecting passlib<1.8,>=1.6.5 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/11/b8/e9a78f3033228013ba8564adad8d0031bf9d39ea3acc3cdb9d55fabeb4ba/passlib-1.7.2-py2.py3-none-any.whl (507 kB) |████████████████████████████████| 507 kB 791 kB/s Collecting hyperframe<6,>=5.1.0 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/19/0c/bf88182bcb5dce3094e2f3e4fe20db28a9928cb7bd5b08024030e4b140db/hyperframe-5.2.0-py2.py3-none-any.whl (12 kB) Collecting pyperclip<1.9,>=1.6.0 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/f6/5b/55866e1cde0f86f5eec59dab5de8a66628cb0d53da74b8dbc15ad8dabda3/pyperclip-1.8.0.tar.gz (16 kB) Collecting protobuf<3.12,>=3.6.0 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/d5/45/c6f7e72311df9d6d28b4c85b4289a2a9b3c2ea69cd180370269e794c123d/protobuf-3.11.3-cp38-cp38-macosx_10_9_x86_64.whl (1.3 MB) |████████████████████████████████| 1.3 MB 4.2 MB/s Collecting ldap3<2.8,>=2.6.1 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/7e/22/c942210f4ad383ccf9e4b0980a2922f0af7452be811c56d585748b9a5a01/ldap3-2.7-py2.py3-none-any.whl (420 kB) |████████████████████████████████| 420 kB 6.1 MB/s Collecting publicsuffix2<3,>=2.20190812 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/9d/16/053c2945c5e3aebeefb4ccd5c5e7639e38bc30ad1bdc7ce86c6d01707726/publicsuffix2-2.20191221-py2.py3-none-any.whl (89 kB) |████████████████████████████████| 89 kB 4.9 MB/s Collecting sortedcontainers<2.2,>=2.1.0 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/13/f3/cf85f7c3a2dbd1a515d51e1f1676d971abe41bba6f4ab5443240d9a78e5b/sortedcontainers-2.1.0-py2.py3-none-any.whl (28 kB) Requirement already satisfied: six>=1.5.2 in /Users/crifanli/.pyenv/versions/3.8.3/lib/python3.8/site-packages (from pyOpenSSL<19.2,>=19.1.0->mitmproxy) (1.15.0) Collecting h11>=0.8.1 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/5a/fd/3dad730b0f95e78aeeb742f96fa7bbecbdd56a58e405d3da440d5bfb90c6/h11-0.9.0-py2.py3-none-any.whl (53 kB) |████████████████████████████████| 53 kB 581 kB/s Collecting cffi!=1.11.3,>=1.8 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/e7/1c/922f632aa04341813b84f30f9b359cf6e8f0d7420a2226604b167643c315/cffi-1.14.0-cp38-cp38-macosx_10_9_x86_64.whl (175 kB) |████████████████████████████████| 175 kB 1.0 MB/s Collecting hpack<4,>=3.0 Using cached https://pypi.tuna.tsinghua.edu.cn/packages/8a/cc/e53517f4a1e13f74776ca93271caef378dadec14d71c61c949d759d3db69/hpack-3.0.0-py2.py3-none-any.whl (38 kB) Requirement already satisfied: Werkzeug>=0.15 in /Users/crifanli/.pyenv/versions/3.8.3/lib/python3.8/site-packages (from flask<1.2,>=1.1.1->mitmproxy) (1.0.1) Requirement already satisfied: itsdangerous>=0.24 in /Users/crifanli/.pyenv/versions/3.8.3/lib/python3.8/site-packages (from flask<1.2,>=1.1.1->mitmproxy) (1.1.0) Requirement already satisfied: Jinja2>=2.10.1 in /Users/crifanli/.pyenv/versions/3.8.3/lib/python3.8/site-packages (from flask<1.2,>=1.1.1->mitmproxy) (2.11.2) Processing /Users/crifanli/Library/Caches/pip/wheels/b9/65/2f/1fc25f1f7c9aa50ad2d3feb3d846ab6964bba70fd06f310287/ruamel.yaml.clib-0.2.0-cp38-cp38-macosx_10_14_x86_64.whl Requirement already satisfied: setuptools in /Users/crifanli/.pyenv/versions/3.8.3/lib/python3.8/site-packages (from protobuf<3.12,>=3.6.0->mitmproxy) (41.2.0) Collecting pycparser Downloading https://pypi.tuna.tsinghua.edu.cn/packages/ae/e7/d9c3a176ca4b02024debf82342dab36efadfc5776f9c8db077e8f6e71821/pycparser-2.20-py2.py3-none-any.whl (112 kB) |████████████████████████████████| 112 kB 1.3 MB/s Requirement already satisfied: MarkupSafe>=0.23 in /Users/crifanli/.pyenv/versions/3.8.3/lib/python3.8/site-packages (from Jinja2>=2.10.1->flask<1.2,>=1.1.1->mitmproxy) (1.1.1) Using legacy setup.py install for tornado, since package 'wheel' is not installed. Using legacy setup.py install for urwid, since package 'wheel' is not installed. Using legacy setup.py install for pyperclip, since package 'wheel' is not installed. Installing collected packages: pycparser, cffi, cryptography, pyOpenSSL, h11, wsproto, tornado, hpack, hyperframe, h2, pyasn1, urwid, zstandard, pyparsing, kaitaistruct, Brotli, ruamel.yaml.clib, ruamel.yaml, blinker, passlib, pyperclip, protobuf, ldap3, publicsuffix2, sortedcontainers, mitmproxy Running setup.py install for tornado ... done Running setup.py install for urwid ... done Running setup.py install for pyperclip ... done Successfully installed Brotli-1.0.7 blinker-1.4 cffi-1.14.0 cryptography-2.9.2 h11-0.9.0 h2-3.2.0 hpack-3.0.0 hyperframe-5.2.0 kaitaistruct-0.8 ldap3-2.7 mitmproxy-5.1.1 passlib-1.7.2 protobuf-3.11.3 publicsuffix2-2.20191221 pyOpenSSL-19.1.0 pyasn1-0.4.8 pycparser-2.20 pyparsing-2.4.7 pyperclip-1.8.0 ruamel.yaml-0.16.10 ruamel.yaml.clib-0.2.0 sortedcontainers-2.1.0 tornado-6.0.4 urwid-2.1.0 wsproto-0.15.0 zstandard-0.13.0
然后去研究看看：
【已解决】pyyaml和ruamel.yaml关系
再看看版本：
crifanli@crifanlideMac ~/dev/DevRoot/appcrawler master ● which mitmdump /Users/crifanli/.pyenv/shims/mitmdump crifanli@crifanlideMac ~/dev/DevRoot/appcrawler master ● which mitmweb /Users/crifanli/.pyenv/shims/mitmweb crifanli@crifanlideMac ~/dev/DevRoot/appcrawler master ● which mitmproxy /Users/crifanli/.pyenv/shims/mitmproxy crifanli@crifanlideMac ~/dev/DevRoot/appcrawler master ● mitmdump --version Mitmproxy: 5.1.1 Python: 3.8.3 OpenSSL: OpenSSL 1.1.1g 21 Apr 2020 Platform: macOS-10.14.4-x86_64-i386-64bit crifanli@crifanlideMac ~/dev/DevRoot/appcrawler master ● mitmproxy --version Mitmproxy: 5.1.1 Python: 3.8.3 OpenSSL: OpenSSL 1.1.1g 21 Apr 2020 Platform: macOS-10.14.4-x86_64-i386-64bit crifanli@crifanlideMac ~/dev/DevRoot/appcrawler master ● mitmweb --version Mitmproxy: 5.1.1 Python: 3.8.3 OpenSSL: OpenSSL 1.1.1g 21 Apr 2020 Platform: macOS-10.14.4-x86_64-i386-64bit
继续
crifanli@crifanlideMac ~/dev/DevRoot/appcrawler master ● mitmdump -k -p 8081 -s middleware/Save1.py sys.executable=/Users/crifanli/.pyenv/versions/3.8.3/bin/python3.8 save url to /Users/crifanli/dev/DevRoot/appcrawler/data/xxx/20200603_xxx_BanMaAICourse/20200603_xxx_BanMaAICourse_app_iOS.txt Loading script middleware/Save1.py Proxy server listening at http://*:8081
貌似可以了。
不过，后续发现，iPhone连不上虚拟机macOS中的代理：
【已解决】和主机Win连同一个WiFi的iPhone无法连接到VMWare中macOS虚拟机中mitmdump代理
至此，代理就没问题了。
【后记】
继续，目前可以继续调试抓包了：
不过data文件暂时还未保存到有效数据，还需进一步确认代理保存url是否真的起效果了。
毕竟刚才mitmdump看到有http地址，data中也没见保存，怀疑代理保存url有问题
算了，停止调试，换其他的试试，能否保存出url
去找找之前调试时，哪个app最能尽快保存出url
对了，必要 app，很快就能出url，换去调试必要：
很明显，没有保存到url
所以，代理还是有点问题的。
去找原因。
感觉像是：
save代码有问题？
去加log试试
/Users/crifanli/dev/DevRoot/appcrawler/middleware/Save1.py
def request(self, flow):
url = flow.request.url
ContentType = self.get_ContentType(str(flow.request.headers))
print("ContentType=%s" % ContentType)
if not url in self.Allurls and not re.search(self.REMOVED, url):
print("to add: url=%s" % url)
self.Allurls.add(url)
print("url=%s" % url)
print("self.DataFilePath=%s" % self.DataFilePath)
with open(self.DataFilePath, "a", encoding="utf-8") as f:
f.write(url + "|" + ContentType)
f.write('\n')去调试看看
不过当改动Save1.py后，mitmdump能立刻检测到，并重新加载，log输出：
sys.executable=/Users/crifanli/.pyenv/versions/3.8.3/bin/python3.8 save url to /Users/crifanli/dev/DevRoot/appcrawler/data/finance/20200526_finance_BiYao/20200526_finance_BiYao_app_iOS.txt Loading script middleware/Save1.py
去调试看看
手动测试后，终于是可以保存url的了：
但是之前为何没有保存，还是很诡异
或许和之前：Mac中也遇到过类似问题，很像。也不知道原因。只是偶尔url过滤失效，重新试试，就又好了。
【总结】
此处win中VMWare中macOS，开启mitmdump代理，过程是：
（1）安装mitmproxy
pip instal pyyaml pip install mitmproxy
（2）VMWare中给macOS虚拟机的网络，从NAT改为 桥接
（3）macOS中启动mitmdump代理
mitmdump -k -p 8081 -s middleware/Save1.py
（4）iPhone中给WiFi设置同网段的macOS的代理
- IP：192.168.31.45
- 端口：8081
注：
（1）关于如何获取macOS当前的IP地址
ifconfig | grep 192.168
得知是：192.168.31.45
（2）虽然iPhone能访问mitmdump代理了
但是Save1.py去保存过滤url，有时候不起效果
->需要手动单独测试一下，确认url保存没问题，才行。