最新消息:20210917 已从crifan.com换到crifan.org

【已解决】Python用requests检测支持已下载文件但是部分文件

Python crifan 439浏览 0评论
折腾:
【未解决】开启和运行另外一个设备id是1的红米Note8Pro的自动化测试安卓项目环境
和:
【未解决】安卓游戏批量测试:VivoGame的21个魔域类游戏20201210
遇到:
由于网络中断等原因,导致下载中断,下载到的安卓apk文件是部分文件。
[201210 12:07:44][DownloadApps.py 139] download com.door.of.magic.vivo/魔法之门Online speed: cur=4.4MB/s, time: total=00:01:07, size: 280.0MB 96.54%
去添加支持已下载,但是是部分已下载。
之前已有逻辑是:
判断apk是否存在,如果存在,则视为已下载。
isDownloaded = CommonUtils.isFileExistAndValid(filepath)

    @staticmethod
    def isFileExistAndValid(filePath):
        isExistFile = os.path.isfile(filePath)
        isValidFile = False
        if isExistFile:
            curFileSize = os.path.getsize(filePath)
            isValidFile = curFileSize > 0
        isExistAndValid = isExistFile and isValidFile
        return isExistAndValid
是不对的。
应该检测已下载的文件,是否是完整的size
如果不是,则 获取已下载大小,接着继续 断点续传
现在去添加此逻辑。
src/common/DownloadApps.py
        totalFileSize = CommonUtils.getFileSizeFromUrl(url)
        totalSizeStr = ""
        if totalFileSize:
            totalSizeStr = CommonUtils.formatSize(totalFileSize)
            logging.info("Get App total size %s from %s", totalSizeStr, url)
        else:
            logging.warning("Fail to get app total size from %s", url)


        isDownloadedAndValid = CommonUtils.isFileExistAndValid(filepath, fullFileSize=totalFileSize)
        if isDownloadedAndValid:
            logging.info("{} is already download".format(appname))
            return
和:
utils/common_utils.py
    @staticmethod
    def getFileSizeFromUrl(fileUrl):
        """Get file size from file url


        Args:
            fileUrl (str): file url
        Returns:
            file size or None
        Raises:
        Examples:
            input: https://gameapktxdl.vivo.com.cn/appstore/developer/soft/20201020/202010201805243ed5v.apk
            output: 154551625
        """
        totalFileSize = None

        try:
            resp = requests.get(fileUrl, stream=True)
            respHeaders = resp.headers # {'Date': 'Thu, 10 Dec 2020 05:27:10 GMT', 'Content-Type': 'application/vnd.android.package-archive', 'Content-Length': '154551625', 'Connection': 'keep-alive', 'Server': 'NWS_TCloud_static_msoc1_xz', 'Cache-Control': 'max-age=600', 'Expires': 'Thu, 10 Dec 2020 05:37:09 GMT', 'Last-Modified': 'Thu, 09 Jan 2020 11:21:35 GMT', 'X-NWS-UUID-VERIFY': '94db2d14f135898d924fb249b13a0964', 'X-Verify-Code': '2871bd7acf67c7e298e9c8d8c865e27d', 'X-NWS-LOG-UUID': 'a83536f2-ab83-465d-ba09-0e19a15cc706', 'X-Cache-Lookup': 'Hit From Disktank3, Hit From Inner Cluster', 'Accept-Ranges': 'bytes', 'ETag': '"46C50A5CADB6BEE339236477BB6DDC14"', 'X-Daa-Tunnel': 'hop_count=2'}
            contentLengthStr = respHeaders['Content-Length'] # '154551625'
            contentLengthInt = int(contentLengthStr) # 154551625
            totalFileSize = contentLengthInt
        except:
            totalFileSize = None


        return totalFileSize # 154551625


    @staticmethod
    def isFileExistAndValid(filePath, fullFileSize=None):
        """Check file exist and valid or not


        Args:
            filePath (str): file path
            fullFileSize (int): full file size
        Returns:
            existed and valid (bool)
        Raises:
        Examples:
        """
        isExistFile = os.path.isfile(filePath)
        isValidFile = False
        if isExistFile:
            curFileSize = os.path.getsize(filePath) # 260900226
            if fullFileSize:
                isValidFile = curFileSize == fullFileSize
            else:
                isValidFile = curFileSize > 0
        isExistAndValid = isExistFile and isValidFile
        return isExistAndValid
去调试看看
【总结】
src/common/DownloadApps.py
    # def download(self, task, taskIndex, taskpath):
    def download(self,task):
        # 下载单个文件
        appname, filepath, url = task
        # '018/永恒纪元', 'task/20201014_fengyun_YongHengJiYuan/20201014_fengyun_YongHengJiYuan_gameApp/018_永恒纪元.apk', 'https://dl.hz.37.com.cn/upload/1_1002464_10068/yonghengjiyuanjie_3384240.apk'


        # # for debug
        # totalFileSize = 447681304
        totalFileSize = CommonUtils.getFileSizeFromUrl(url) # 154551625
        totalSizeStr = ""
        if totalFileSize:
            totalSizeStr = CommonUtils.formatSize(totalFileSize)
            logging.info("Get app total size %s from %s", totalSizeStr, url)
        else:
            logging.warning("Fail to get app total size from %s", url)


        isDownloadedAndValid = CommonUtils.isFileExistAndValid(filepath, fullFileSize=totalFileSize)
        if isDownloadedAndValid:
            logging.info("{} is already download".format(appname))
            return


        logging.info("start to download {0},url {1}".format(appname, url))
        downloadCalcTimeKey = "downloadAndroidApk_%s" % appname # 'downloadAndroidApk_018/永恒纪元'
        CommonUtils.calcTimeStart(downloadCalcTimeKey)


        totalDownloadedSize = os.path.getsize(filepath) if os.path.exists(filepath) else 0
        if totalFileSize:
            totalDownloadedPercent = float(totalDownloadedSize / totalFileSize) # 293601280 / 304130341 = 0.9653797744566367
        else:
            totalDownloadedPercent = 0.0
        totalDownloadedPercent100 = totalDownloadedPercent * 100
        logging.info("Has downloaded %.2f%%", totalDownloadedPercent100)
        headers = {
            'Range': 'bytes=%d-' % totalDownloadedSize,
            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.81 Safari/537.36",
        }
        r = requests.get(url, stream=True, headers=headers)
        # ChunkSize = 1024*1024*10 # 10MB
        ChunkSize = 1024*1024*20 # 20MB
        with open(filepath, "ab") as f:
            startTime = time.time()
            prevTime = startTime
            for chunkBytes in r.iter_content(chunk_size=ChunkSize):
                if chunkBytes:
                    curDownloadedSize = len(chunkBytes) # 10485760
                    curTime = time.time() # 1606456020.0718982


                    totalDownloadedSize += curDownloadedSize # 10485760
                    f.write(chunkBytes)
                    f.flush()


                    if totalFileSize:
                        downloadedPercent = round(100 * totalDownloadedSize / totalFileSize, 2) # 0.7
                    else:
                        downloadedPercent = "Unknown"


                    downloadedSizeStr = CommonUtils.formatSize(totalDownloadedSize) # '10.0MB'


                    curDownloadTime = curTime - prevTime # 15.63818907737732
                    curSpeed = curDownloadedSize / curDownloadTime # 670522.651191692
                    curSpeedStr = CommonUtils.formatSize(curSpeed) # '654.8KB'


                    totalDownloadTime = curTime - startTime # 15.63818907737732
                    averageSpeed = totalDownloadedSize / totalDownloadTime # 670522.651191692
                    averageSpeedStr = CommonUtils.formatSize(averageSpeed) # '654.8KB'


                    totalDownloadTimeDict = CommonUtils.floatSecondsToDatetimeDict(totalDownloadTime)
                    totalDownloadTimeStr = CommonUtils.datetimeDictToStr(totalDownloadTimeDict, isShowMilliSecPart=False)


                    logging.info("download %s speed: cur=%s/s, time: total=%s, size: %s %s%%",
                        appname, curSpeedStr, totalDownloadTimeStr, downloadedSizeStr, downloadedPercent)


                    prevTime = curTime


        downloadTimeSecondsFloat = CommonUtils.calcTimeEnd(downloadCalcTimeKey) # 259.60212874412537
        downloadTimeDict = CommonUtils.floatSecondsToDatetimeDict(downloadTimeSecondsFloat) # {'days': 0, 'hours': 0, 'microseconds': 128, 'millseconds': 602, 'minutes': 4, 'seconds': 19}
        del downloadTimeDict["millseconds"]
        del downloadTimeDict["microseconds"]
        downloadTimeStr = CommonUtils.datetimeDictToStr(downloadTimeDict) # '00:04:19'
        if totalFileSize:
            speedFloat = totalFileSize / downloadTimeSecondsFloat # 1724490.1117172781
            speedStr = "%s/s" % CommonUtils.formatSize(speedFloat) # '1.6MB/s'
        else:
            speedStr = "Unknown"
        logging.info("End download %s, size=%s, time=%s, speed=%s", appname, totalSizeStr, downloadTimeStr, speedStr)
和:
utils/common_utils.py
    @staticmethod
    def getFileSizeFromUrl(fileUrl):
        """Get file size from file url


        Args:
            fileUrl (str): file url
        Returns:
            file size or None
        Raises:
        Examples:
            input: https://gameapktxdl.vivo.com.cn/appstore/developer/soft/20201020/202010201805243ed5v.apk
            output: 154551625
        """
        totalFileSize = None


        try:
            resp = requests.get(fileUrl, stream=True)
            respHeaders = resp.headers # {'Date': 'Thu, 10 Dec 2020 05:27:10 GMT', 'Content-Type': 'application/vnd.android.package-archive', 'Content-Length': '154551625', 'Connection': 'keep-alive', 'Server': 'NWS_TCloud_static_msoc1_xz', 'Cache-Control': 'max-age=600', 'Expires': 'Thu, 10 Dec 2020 05:37:09 GMT', 'Last-Modified': 'Thu, 09 Jan 2020 11:21:35 GMT', 'X-NWS-UUID-VERIFY': '94db2d14f135898d924fb249b13a0964', 'X-Verify-Code': '2871bd7acf67c7e298e9c8d8c865e27d', 'X-NWS-LOG-UUID': 'a83536f2-ab83-465d-ba09-0e19a15cc706', 'X-Cache-Lookup': 'Hit From Disktank3, Hit From Inner Cluster', 'Accept-Ranges': 'bytes', 'ETag': '"46C50A5CADB6BEE339236477BB6DDC14"', 'X-Daa-Tunnel': 'hop_count=2'}
            contentLengthStr = respHeaders['Content-Length'] # '154551625'
            contentLengthInt = int(contentLengthStr) # 154551625
            totalFileSize = contentLengthInt
        except:
            totalFileSize = None


        return totalFileSize # 154551625


    @staticmethod
    def isFileExistAndValid(filePath, fullFileSize=None):
        """Check file exist and valid or not


        Args:
            filePath (str): file path
            fullFileSize (int): full file size
        Returns:
            existed and valid (bool)
        Raises:
        Examples:
        """
        isExistFile = os.path.isfile(filePath)
        isValidFile = False
        if isExistFile:
            curFileSize = os.path.getsize(filePath) # 154551625
            if fullFileSize:
                isValidFile = curFileSize == fullFileSize
            else:
                isValidFile = curFileSize > 0
        isExistAndValid = isExistFile and isValidFile
        return isExistAndValid
输出:
[201210 13:34:38][DownloadApps.py 88 ] start to download com.door.of.magic.vivo/魔法之门Online,url https://gameapktxdl.vivo.com.cn/appstore/developer/soft/20201020/202010201805243ed5v.apk
[201210 13:34:42][DownloadApps.py 98 ] Has downloaded 96.54%
[201210 13:34:58][DownloadApps.py 136] download com.door.of.magic.vivo/魔法之门Online speed: cur=3.9MB/s, time: total=00:00:02, size: 290.0MB 100.0%
[201210 13:35:09][DownloadApps.py 151] End download com.door.of.magic.vivo/魔法之门Online, size=290.0MB, time=00:00:19, speed=14.7MB/s

转载请注明:在路上 » 【已解决】Python用requests检测支持已下载文件但是部分文件

发表我的评论
取消评论

表情

Hi,您需要填写昵称和邮箱!

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址
92 queries in 0.236 seconds, using 23.39MB memory