折腾：
【未解决】其他下载文件速度更快的Python的库
期间，看到一个pget：
号称可以多线程下载文件，希望可以提高下载速度。
去测pget：
~/dev/crifan/python/downlodSpeedTest pip install pget Collecting pget Downloading PGet-0.5.0.tar.gz (6.9 kB) Collecting future Using cached future-0.18.2.tar.gz (829 kB) Requirement already satisfied: requests in /Users/xxx/.pyenv/versions/3.6.5/lib/python3.6/site-packages (from pget) (2.22.0) Collecting wheel Using cached wheel-0.35.1-py2.py3-none-any.whl (33 kB) Requirement already satisfied: idna<2.9,>=2.5 in /Users/xxx/.pyenv/versions/3.6.5/lib/python3.6/site-packages (from requests->pget) (2.8) Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /Users/xxx/.pyenv/versions/3.6.5/lib/python3.6/site-packages (from requests->pget) (1.25.7) Requirement already satisfied: certifi>=2017.4.17 in /Users/xxx/.pyenv/versions/3.6.5/lib/python3.6/site-packages (from requests->pget) (2019.11.28) Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /Users/xxx/.pyenv/versions/3.6.5/lib/python3.6/site-packages (from requests->pget) (3.0.4) Using legacy 'setup.py install' for pget, since package 'wheel' is not installed. Using legacy 'setup.py install' for future, since package 'wheel' is not installed. Installing collected packages: future, wheel, pget Running setup.py install for future ... done Running setup.py install for pget ... done Successfully installed future-0.18.2 pget-0.5.0 wheel-0.35.1
再去写代码测试
pget的实例代码很挫，都没解释清楚各个参数如何写
只能自己去看代码了。
pget_code/pget/pget/bin.py
parser.add_argument('-C', '--chunks', dest='chunks', type=int, default=8, help='Parallel connection count')看来是 多线程个数 的意思？
另外还有：
parser.add_argument('-F','--highspeed', dest='high_speed', default=False,
const=True, help='High speed connection flag', action='store_const')或许可以加上 高速？
先去试试：
from pget.down import Downloader
fileUrl = "https://gameapktxdl.vivo.com.cn/appstore/developer/soft/20201013/202010131747351e59m.apk" # 1.6GB
downloadedFilename = "202010131747351e59m.apk"
downloadedFullPath = os.path.join("output", downloadedFilename)
downloader = Downloader(
fileUrl,
downloadedFullPath,
chunk_count=4,
# high_speed=True
)看看下载速度如何
结果代码：
from pget.down import Downloader
fileUrl = "https://gameapktxdl.vivo.com.cn/appstore/developer/soft/20201013/202010131747351e59m.apk" # 1.6GB
downloadedFilename = "202010131747351e59m.apk"
downloadedFullPath = os.path.join("output", downloadedFilename) # 'output/202010131747351e59m.apk'
downloader = Downloader(
fileUrl,
downloadedFullPath,
chunk_count=4,
# high_speed=True
)
print("downloader=%s" % downloader) # downloader=<pget.down.Downloader object at 0x104fc5490>
def downloadCallback(downloadedSize):
print("downloadedSize=%s" % downloadedSize)
downloader.start()
callback_threshold = 10*1024*1024 # 10MB
downloader.subscribe(downloadCallback, callback_threshold)
downloader.wait_for_finish()
print("Complete download %s to %s" % (fileUrl, downloadedFullPath))始终没有调用到callback
感觉不对。
减少chunk的size：
callback_threshold = 1*1024*1024 # 1MB
重新调试看看，
过了几十分钟后，竟然可以有callback了：
去看看这个last_total的size：
1768171236 = 1.65G
好像就是 全部下载后，才有callback
然后最后却多次输出callback
downloadedSize=<pget.down.Downloader object at 0x10900b070> downloadedSize=<pget.down.Downloader object at 0x10900b070> downloadedSize=<pget.down.Downloader object at 0x10900b070>
。。。
而下载到的这个文件，中间被我调试中断后，就没有完全输出，只有部分大小：
所以，代码逻辑很是有问题。
看起代码：
pget_code/pget/pget/chunk.py
if self.high_speed: self.download_iter_size = 1024*512 # Half a megabyte else: self.download_iter_size = 1024 # a kilobyte
貌似只是chunk size大小不同而已。。。
1K 和 512KB 的区别。。。
另外，关于chunk
pget_code/pget/pget/down.py
if self.chunk_count == 0: chunk_file = tempfile.TemporaryFile() new_chunk = Chunk(self, self.url, file=chunk_file, high_speed=self.high_speed, headers=self.headers) self.__chunks.append(new_chunk) new_chunk.start() else: chunk_size = self.total_length / self.chunk_count for chunk_number in range(self.chunk_count): chunk_file = tempfile.TemporaryFile() if chunk_number != self.chunk_count - 1: new_chunk = Chunk( self, self.url, chunk_file, start_byte=chunk_number * chunk_size, end_byte=((chunk_number + 1) * chunk_size) - 1, number=chunk_number, high_speed=self.high_speed, headers=self.headers) else: new_chunk = Chunk( self, self.url, chunk_file, start_byte=chunk_number * chunk_size, end_byte=self.total_length - 1, number=chunk_number, high_speed=self.high_speed, headers=self.headers)
感觉像是：
此处chunk是：
分多个线程，每个线程去下载的大小？
pget_code/pget/pget/bin.py
downloader.subscribe(download_callback, 256) downloader.start() signal.signal(signal.SIGTERM, handler) signal.signal(signal.SIGINT, handler) downloader.wait_for_finish()
也是
先有subscribe
再去start
符合正常逻辑
而之前实例代码就是反的：
先start
再subscribe
当时就觉得有问题。
那改为：
callback_threshold = 10*1024*1024 # 10MB # callback_threshold = 1*1024*1024 # 1MB downloader.subscribe(downloadCallback, callback_threshold) downloader.start() downloader.wait_for_finish()
结果：
还是没立刻得到callback
加上：
high_speed=True
结果：
影响当前下载
[201127 16:15:32][DownloadApps.py 139] download snggzl/少年三国志：零 speed: cur=1.7MB/s, time: total=00:09:22, size: 920.0MB 76.25% [201127 16:15:47][DownloadApps.py 139] download snggzl/少年三国志：零 speed: cur=707.4KB/s, time: total=00:09:36, size: 930.0MB 77.07% [201127 16:16:08][DownloadApps.py 139] download snggzl/少年三国志：零 speed: cur=497.1KB/s, time: total=00:09:57, size: 940.0MB 77.9% [201127 16:16:15][DownloadApps.py 139] download snggzl/少年三国志：零 speed: cur=1.3MB/s, time: total=00:10:04, size: 950.0MB 78.73%
抽空再测
看了下代码，貌似的确是 多线程：
pget_code/pget/pget/down.py
。。。 self.__chunks.append(new_chunk) new_chunk.start() speed_thread = threading.Thread(target=self.speed_func) speed_thread.start() for chunk in self.__chunks: chunk.thread.join() if self.__state == Downloader.STOPPED: return # Forcefully update subscribers for last time. self.notify_subs(True) self.__state = Downloader.MERGING speed_thread.join() 。。。
那就去：
暂时忽略subscribe
看看总体下载速度是否很快
downloader=<pget.down.Downloader object at 0x108848040> Complete download https://gameapktxdl.vivo.com.cn/appstore/developer/soft/20201013/202010131747351e59m.apk to output/202010131747351e59m.apk downloadSpeedStr=1.2MB
-》即 最后平均速度也只有 1.2MB/s
-》比requests的1.7MB/s 更低
放弃。
转载请注明：在路上 » 【未解决】Mac中尝试用pget加速文件下载提高下载速度