最新消息:20210917 已从crifan.com换到crifan.org

【已解决】给VMWare中macOS中抓包项目开启mitmdump代理

macOS crifan 503浏览 0评论
折腾:
【未解决】Win中VMWare中macOS中调试抓包项目
期间,调试
【未解决】Win中尝试访问VMWare中macOS中8100端口的wda服务
期间发现:
iOS中app无法访问网络了:
网络异常,请检查网络设置
请检查您当前的网络环境,如果其他App可以正常使用,请到设置-》斑马AI可-无线数据中允许斑马AI课访问网络。检查后,点击重试按钮。
去看了看,果然是网络问题:
当前WiFi网络加了代理:
但是当前忘了开启代理了。
所以去开启mitmdump代理
crifanli@crifanlideMac  ~/dev/DevRoot/appcrawler   master ●  pwd
/Users/crifanli/dev/DevRoot/appcrawler
crifanli@crifanlideMac  ~/dev/DevRoot/appcrawler   master ●  ll
total 120
-rw-r--r--   1 crifanli  staff    27K  6 23 16:20 README.md
-rw-r--r--   1 crifanli  staff   2.4K  6 23 09:14 analyzeStart.py
-rw-r--r--   1 crifanli  staff   6.2K  6 23 16:20 crawlerStart.py
drwxr-xr-x   3 crifanli  staff    96B  6 28 10:48 data
drwxr-xr-x   2 crifanli  staff    64B  6 28 16:29 debug
drwxr-xr-x   7 crifanli  staff   224B  6 23 09:14 gameAutomation
drwxr-xr-x   5 crifanli  staff   160B  6 23 09:14 iOSAutomation
drwxr-xr-x   3 crifanli  staff    96B  6 23 16:29 logging
drwxr-xr-x  24 crifanli  staff   768B  6 23 16:20 middleware
drwxr-xr-x   7 crifanli  staff   224B  6 23 16:20 platformIntegration
drwxr-xr-x  13 crifanli  staff   416B  6 23 16:20 process
-rw-r--r--   1 crifanli  staff   2.6K  6 23 09:14 queryService.py
-rw-r--r--   1 crifanli  staff   2.6K  6 23 09:14 queryService1.py
-rw-r--r--   1 crifanli  staff   4.5K  6 23 15:32 requirements.txt
drwxr-xr-x   7 crifanli  staff   224B  6 23 16:26 service
drwxr-xr-x   8 crifanli  staff   256B  6 24 13:54 src
-rw-r--r--   1 crifanli  staff    72B  6 23 09:14 stop_mongod.js
drwxr-xr-x  17 crifanli  staff   544B  6 24 11:26 task
drwxr-xr-x   5 crifanli  staff   160B  6 23 09:14 tmp
drwxr-xr-x  17 crifanli  staff   544B  6 23 09:14 tool
drwxr-xr-x   6 crifanli  staff   192B  6 23 16:26 utils
drwxr-xr-x   6 crifanli  staff   192B  6 23 11:46 venv
crifanli@crifanlideMac  ~/dev/DevRoot/appcrawler   master ●  mitmdump -k -p 8081 -s middleware/Save1.py
zsh: command not found: mitmdump
很明显,macOS系统全局还没安装mitmproxy呢。去安装。
虽然好像直接brew即可,不过也去找之前帖子,参考看看。
【已解决】Mac中安装Mitmdump和启动服务
pip install mitmproxy
而不用brew安装,否则后续会出现yaml找不到的问题
不过也要先去macOS中安装用pip安装yaml,所以应该是
pip instal pyyaml
pip install mitmproxy
详细log
✘ crifanli@crifanlideMac  ~/dev/DevRoot/appcrawler   master ●  pip install pyyaml
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Processing /Users/crifanli/Library/Caches/pip/wheels/48/41/22/8d283b016a1f9f8eac6d9f3c5b99cb58aabf8681225ab4f6dd/PyYAML-5.3.1-cp38-cp38-macosx_10_14_x86_64.whl
Installing collected packages: pyyaml
Successfully installed pyyaml-5.3.1
以及:
crifanli@crifanlideMac  ~/dev/DevRoot/appcrawler   master ●  pip install mitmproxy
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Collecting mitmproxy
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/d3/b9/fdfa51a2ae6ab968068a1b682e0c9efb76cc2f0493cf3e4fab33166d8345/mitmproxy-5.1.1-py3-none-any.whl (1.3 MB)
     |████████████████████████████████| 1.3 MB 1.3 MB/s
Collecting pyOpenSSL<19.2,>=19.1.0
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/9e/de/f8342b68fa9e981d348039954657bdf681b2ab93de27443be51865ffa310/pyOpenSSL-19.1.0-py2.py3-none-any.whl (53 kB)
     |████████████████████████████████| 53 kB 5.3 MB/s
Collecting wsproto<0.16,>=0.14
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/d5/8b/96575a9a73591ce8e5e519375e30565bf9d299d0ced98a2970c225abedf4/wsproto-0.15.0-py2.py3-none-any.whl (24 kB)
Collecting tornado<7,>=4.3
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/95/84/119a46d494f008969bf0c775cb2c6b3579d3c4cc1bb1b41a022aa93ee242/tornado-6.0.4.tar.gz (496 kB)
     |████████████████████████████████| 496 kB 11.2 MB/s
Collecting cryptography<3.0,>=2.9
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/0f/ed/93c2217e732e75c21477c350b2d10a77e08fe2e0065c14644bdb110acfdd/cryptography-2.9.2-cp35-abi3-macosx_10_9_x86_64.whl (1.8 MB)
     |████████████████████████████████| 1.8 MB 30.7 MB/s
Collecting h2<4,>=3.2.0
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/25/de/da019bcc539eeab02f6d45836f23858ac467f584bfec7a526ef200242afe/h2-3.2.0-py2.py3-none-any.whl (65 kB)
     |████████████████████████████████| 65 kB 5.5 MB/s
Requirement already satisfied: click<8,>=7.0 in /Users/crifanli/.pyenv/versions/3.8.3/lib/python3.8/site-packages (from mitmproxy) (7.1.2)
Collecting pyasn1<0.5,>=0.3.1
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/62/1e/a94a8d635fa3ce4cfc7f506003548d0a2447ae76fd5ca53932970fe3053f/pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
     |████████████████████████████████| 77 kB 142 kB/s
Collecting urwid<2.2,>=2.1.0
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/45/dd/d57924f77b0914f8a61c81222647888fbb583f89168a376ffeb5613b02a6/urwid-2.1.0.tar.gz (630 kB)
     |████████████████████████████████| 630 kB 691 kB/s
Collecting zstandard<0.14,>=0.11
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/c6/79/a7b31258dfc071344e5e21976a395310330e72ffad0e3d12a68e3d2c82fe/zstandard-0.13.0-cp38-cp38-macosx_10_9_x86_64.whl (460 kB)
     |████████████████████████████████| 460 kB 1.1 MB/s
Requirement already satisfied: flask<1.2,>=1.1.1 in /Users/crifanli/.pyenv/versions/3.8.3/lib/python3.8/site-packages (from mitmproxy) (1.1.2)
Collecting pyparsing<2.5,>=2.4.2
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/8a/bb/488841f56197b13700afd5658fc279a2025a39e22449b7cf29864669b15d/pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
     |████████████████████████████████| 67 kB 3.3 MB/s
Requirement already satisfied: certifi>=2019.9.11 in /Users/crifanli/.pyenv/versions/3.8.3/lib/python3.8/site-packages (from mitmproxy) (2020.6.20)
Processing /Users/crifanli/Library/Caches/pip/wheels/8c/3c/0a/2b4abe85e05a2c31ef23e71e79d7e8c20b1c42dc56b52912c6/kaitaistruct-0.8-py2.py3-none-any.whl
Collecting Brotli<1.1,>=1.0
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/16/f8/88122cbd692b56f8bdb9e16b6bb7aad6cda2a6057d28dcaa1a409b6aa0a6/Brotli-1.0.7-cp38-cp38-macosx_10_9_x86_64.whl (412 kB)
     |████████████████████████████████| 412 kB 257 kB/s
Collecting ruamel.yaml<0.17,>=0.16
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/a6/92/59af3e38227b9cc14520bf1e59516d99ceca53e3b8448094248171e9432b/ruamel.yaml-0.16.10-py2.py3-none-any.whl (111 kB)
     |████████████████████████████████| 111 kB 3.8 MB/s
Processing /Users/crifanli/Library/Caches/pip/wheels/22/81/56/0ae6d0abd812c969a7bb0a56e80667d5f0d6aa57d01bd3fe1d/blinker-1.4-py3-none-any.whl
Collecting passlib<1.8,>=1.6.5
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/11/b8/e9a78f3033228013ba8564adad8d0031bf9d39ea3acc3cdb9d55fabeb4ba/passlib-1.7.2-py2.py3-none-any.whl (507 kB)
     |████████████████████████████████| 507 kB 791 kB/s
Collecting hyperframe<6,>=5.1.0
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/19/0c/bf88182bcb5dce3094e2f3e4fe20db28a9928cb7bd5b08024030e4b140db/hyperframe-5.2.0-py2.py3-none-any.whl (12 kB)
Collecting pyperclip<1.9,>=1.6.0
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/f6/5b/55866e1cde0f86f5eec59dab5de8a66628cb0d53da74b8dbc15ad8dabda3/pyperclip-1.8.0.tar.gz (16 kB)
Collecting protobuf<3.12,>=3.6.0
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/d5/45/c6f7e72311df9d6d28b4c85b4289a2a9b3c2ea69cd180370269e794c123d/protobuf-3.11.3-cp38-cp38-macosx_10_9_x86_64.whl (1.3 MB)
     |████████████████████████████████| 1.3 MB 4.2 MB/s
Collecting ldap3<2.8,>=2.6.1
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/7e/22/c942210f4ad383ccf9e4b0980a2922f0af7452be811c56d585748b9a5a01/ldap3-2.7-py2.py3-none-any.whl (420 kB)
     |████████████████████████████████| 420 kB 6.1 MB/s
Collecting publicsuffix2<3,>=2.20190812
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/9d/16/053c2945c5e3aebeefb4ccd5c5e7639e38bc30ad1bdc7ce86c6d01707726/publicsuffix2-2.20191221-py2.py3-none-any.whl (89 kB)
     |████████████████████████████████| 89 kB 4.9 MB/s
Collecting sortedcontainers<2.2,>=2.1.0
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/13/f3/cf85f7c3a2dbd1a515d51e1f1676d971abe41bba6f4ab5443240d9a78e5b/sortedcontainers-2.1.0-py2.py3-none-any.whl (28 kB)
Requirement already satisfied: six>=1.5.2 in /Users/crifanli/.pyenv/versions/3.8.3/lib/python3.8/site-packages (from pyOpenSSL<19.2,>=19.1.0->mitmproxy) (1.15.0)
Collecting h11>=0.8.1
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/5a/fd/3dad730b0f95e78aeeb742f96fa7bbecbdd56a58e405d3da440d5bfb90c6/h11-0.9.0-py2.py3-none-any.whl (53 kB)
     |████████████████████████████████| 53 kB 581 kB/s
Collecting cffi!=1.11.3,>=1.8
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/e7/1c/922f632aa04341813b84f30f9b359cf6e8f0d7420a2226604b167643c315/cffi-1.14.0-cp38-cp38-macosx_10_9_x86_64.whl (175 kB)
     |████████████████████████████████| 175 kB 1.0 MB/s
Collecting hpack<4,>=3.0
  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/8a/cc/e53517f4a1e13f74776ca93271caef378dadec14d71c61c949d759d3db69/hpack-3.0.0-py2.py3-none-any.whl (38 kB)
Requirement already satisfied: Werkzeug>=0.15 in /Users/crifanli/.pyenv/versions/3.8.3/lib/python3.8/site-packages (from flask<1.2,>=1.1.1->mitmproxy) (1.0.1)
Requirement already satisfied: itsdangerous>=0.24 in /Users/crifanli/.pyenv/versions/3.8.3/lib/python3.8/site-packages (from flask<1.2,>=1.1.1->mitmproxy) (1.1.0)
Requirement already satisfied: Jinja2>=2.10.1 in /Users/crifanli/.pyenv/versions/3.8.3/lib/python3.8/site-packages (from flask<1.2,>=1.1.1->mitmproxy) (2.11.2)
Processing /Users/crifanli/Library/Caches/pip/wheels/b9/65/2f/1fc25f1f7c9aa50ad2d3feb3d846ab6964bba70fd06f310287/ruamel.yaml.clib-0.2.0-cp38-cp38-macosx_10_14_x86_64.whl
Requirement already satisfied: setuptools in /Users/crifanli/.pyenv/versions/3.8.3/lib/python3.8/site-packages (from protobuf<3.12,>=3.6.0->mitmproxy) (41.2.0)
Collecting pycparser
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/ae/e7/d9c3a176ca4b02024debf82342dab36efadfc5776f9c8db077e8f6e71821/pycparser-2.20-py2.py3-none-any.whl (112 kB)
     |████████████████████████████████| 112 kB 1.3 MB/s
Requirement already satisfied: MarkupSafe>=0.23 in /Users/crifanli/.pyenv/versions/3.8.3/lib/python3.8/site-packages (from Jinja2>=2.10.1->flask<1.2,>=1.1.1->mitmproxy) (1.1.1)
Using legacy setup.py install for tornado, since package 'wheel' is not installed.
Using legacy setup.py install for urwid, since package 'wheel' is not installed.
Using legacy setup.py install for pyperclip, since package 'wheel' is not installed.
Installing collected packages: pycparser, cffi, cryptography, pyOpenSSL, h11, wsproto, tornado, hpack, hyperframe, h2, pyasn1, urwid, zstandard, pyparsing, kaitaistruct, Brotli, ruamel.yaml.clib, ruamel.yaml, blinker, passlib, pyperclip, protobuf, ldap3, publicsuffix2, sortedcontainers, mitmproxy
    Running setup.py install for tornado ... done
    Running setup.py install for urwid ... done
    Running setup.py install for pyperclip ... done
Successfully installed Brotli-1.0.7 blinker-1.4 cffi-1.14.0 cryptography-2.9.2 h11-0.9.0 h2-3.2.0 hpack-3.0.0 hyperframe-5.2.0 kaitaistruct-0.8 ldap3-2.7 mitmproxy-5.1.1 passlib-1.7.2 protobuf-3.11.3 publicsuffix2-2.20191221 pyOpenSSL-19.1.0 pyasn1-0.4.8 pycparser-2.20 pyparsing-2.4.7 pyperclip-1.8.0 ruamel.yaml-0.16.10 ruamel.yaml.clib-0.2.0 sortedcontainers-2.1.0 tornado-6.0.4 urwid-2.1.0 wsproto-0.15.0 zstandard-0.13.0
然后去研究看看:
【已解决】pyyaml和ruamel.yaml关系
再看看版本:
crifanli@crifanlideMac  ~/dev/DevRoot/appcrawler   master ●  which mitmdump
/Users/crifanli/.pyenv/shims/mitmdump
crifanli@crifanlideMac  ~/dev/DevRoot/appcrawler   master ●  which mitmweb
/Users/crifanli/.pyenv/shims/mitmweb
crifanli@crifanlideMac  ~/dev/DevRoot/appcrawler   master ●  which mitmproxy
/Users/crifanli/.pyenv/shims/mitmproxy
crifanli@crifanlideMac  ~/dev/DevRoot/appcrawler   master ●  mitmdump --version
Mitmproxy: 5.1.1
Python:    3.8.3
OpenSSL:   OpenSSL 1.1.1g  21 Apr 2020
Platform:  macOS-10.14.4-x86_64-i386-64bit
crifanli@crifanlideMac  ~/dev/DevRoot/appcrawler   master ●  mitmproxy --version
Mitmproxy: 5.1.1
Python:    3.8.3
OpenSSL:   OpenSSL 1.1.1g  21 Apr 2020
Platform:  macOS-10.14.4-x86_64-i386-64bit
crifanli@crifanlideMac  ~/dev/DevRoot/appcrawler   master ●  mitmweb --version
Mitmproxy: 5.1.1
Python:    3.8.3
OpenSSL:   OpenSSL 1.1.1g  21 Apr 2020
Platform:  macOS-10.14.4-x86_64-i386-64bit
继续
crifanli@crifanlideMac  ~/dev/DevRoot/appcrawler   master ●  mitmdump -k -p 8081 -s middleware/Save1.py
sys.executable=/Users/crifanli/.pyenv/versions/3.8.3/bin/python3.8
save url to /Users/crifanli/dev/DevRoot/appcrawler/data/xxx/20200603_xxx_BanMaAICourse/20200603_xxx_BanMaAICourse_app_iOS.txt
Loading script middleware/Save1.py
Proxy server listening at http://*:8081
貌似可以了。
不过,后续发现,iPhone连不上虚拟机macOS中的代理:
【已解决】和主机Win连同一个WiFi的iPhone无法连接到VMWare中macOS虚拟机中mitmdump代理
至此,代理就没问题了。
【后记】
继续,目前可以继续调试抓包了:
不过data文件暂时还未保存到有效数据,还需进一步确认代理保存url是否真的起效果了。
毕竟刚才mitmdump看到有http地址,data中也没见保存,怀疑代理保存url有问题
算了,停止调试,换其他的试试,能否保存出url
去找找之前调试时,哪个app最能尽快保存出url
对了,必要 app,很快就能出url,换去调试必要:
很明显,没有保存到url
所以,代理还是有点问题的。
去找原因。
感觉像是:
save代码有问题?
去加log试试
/Users/crifanli/dev/DevRoot/appcrawler/middleware/Save1.py
    def request(self, flow):
        url = flow.request.url
        ContentType = self.get_ContentType(str(flow.request.headers))
        print("ContentType=%s" % ContentType)
        if not url in self.Allurls and not re.search(self.REMOVED, url):
            print("to add: url=%s" % url)
            self.Allurls.add(url)
            print("url=%s" % url)
            print("self.DataFilePath=%s" % self.DataFilePath)
            with open(self.DataFilePath, "a", encoding="utf-8") as f:
                f.write(url + "|" + ContentType)
                f.write('\n')
去调试看看
不过当改动Save1.py后,mitmdump能立刻检测到,并重新加载,log输出:
sys.executable=/Users/crifanli/.pyenv/versions/3.8.3/bin/python3.8
save url to /Users/crifanli/dev/DevRoot/appcrawler/data/finance/20200526_finance_BiYao/20200526_finance_BiYao_app_iOS.txt
Loading script middleware/Save1.py
去调试看看
手动测试后,终于是可以保存url的了:
但是之前为何没有保存,还是很诡异
或许和之前:Mac中也遇到过类似问题,很像。也不知道原因。只是偶尔url过滤失效,重新试试,就又好了。
【总结】
此处win中VMWare中macOS,开启mitmdump代理,过程是:
(1)安装mitmproxy
pip instal pyyaml
pip install mitmproxy
(2)VMWare中给macOS虚拟机的网络,从NAT改为 桥接
(3)macOS中启动mitmdump代理
mitmdump -k -p 8081 -s middleware/Save1.py
(4)iPhone中给WiFi设置同网段的macOS的代理
  • IP:192.168.31.45
  • 端口:8081
注:
(1)关于如何获取macOS当前的IP地址
ifconfig | grep 192.168
得知是:192.168.31.45
(2)虽然iPhone能访问mitmdump代理了
但是Save1.py去保存过滤url,有时候不起效果
->需要手动单独测试一下,确认url保存没问题,才行。

转载请注明:在路上 » 【已解决】给VMWare中macOS中抓包项目开启mitmdump代理

发表我的评论
取消评论

表情

Hi,您需要填写昵称和邮箱!

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址
90 queries in 0.203 seconds, using 23.35MB memory