折腾:
【未解决】用蓝图和工厂模式去优化现有Flask项目代码结构
期间,对于Flask的app的工厂模式的create_app,去初始化时,已经解决了Celery的工厂模式问题:
【已解决】Flask中如何用工厂模式初始化Celery
但是想要完整调试项目,需要Mac的本地终端中去运行celery的worker,结果出错:

➜ xxxRobotDemoServer git:(master) ✗ celery worker -A resources.celery_task.celery --loglevel=DEBUG Traceback (most recent call last): File "/Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/bin/celery", line 11, in <module> sys.exit(main()) File "/Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages/celery/__main__.py", line 16, in main _main() File "/Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages/celery/bin/celery.py", line 322, in main cmd.execute_from_commandline(argv) File "/Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages/celery/bin/celery.py", line 496, in execute_from_commandline super(CeleryCommand, self).execute_from_commandline(argv))) File "/Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages/celery/bin/base.py", line 273, in execute_from_commandline argv = self.setup_app_from_commandline(argv) File "/Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages/celery/bin/base.py", line 479, in setup_app_from_commandline self.app = self.find_app(app) File "/Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages/celery/bin/base.py", line 501, in find_app return find_app(app, symbol_by_name=self.symbol_by_name) File "/Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages/celery/app/utils.py", line 359, in find_app sym = symbol_by_name(app, imp=imp) File "/Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages/celery/bin/base.py", line 504, in symbol_by_name return imports.symbol_by_name(name, imp=imp) File "/Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages/kombu/utils/imports.py", line 56, in symbol_by_name module = imp(module_name, package=package, **kwargs) File "/Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages/celery/utils/imports.py", line 104, in import_from_cwd return imp(module, package=package) File "/Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/importlib/__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 994, in _gcd_import File "<frozen importlib._bootstrap>", line 971, in _find_and_load File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 665, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 678, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/Users/crifan/dev/dev_root/company/xxx/projects/robotDemo/server/xxxRobotDemoServer/resources/celery_task.py", line 7, in <module> from resources.tts import refreshAzureSpeechToken File "/Users/crifan/dev/dev_root/company/xxx/projects/robotDemo/server/xxxRobotDemoServer/resources/tts.py", line 9, in <module> app = g.app File "/Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages/werkzeug/local.py", line 347, in __getattr__ return getattr(self._get_current_object(), name) File "/Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages/werkzeug/local.py", line 306, in _get_current_object return self.__local() File "/Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages/flask/globals.py", line 44, in _lookup_app_object raise RuntimeError(_app_ctx_err_msg) RuntimeError: Working outside of application context. This typically means that you attempted to use functionality that needed to interface with the current application object in some way. To solve this, set up an application context with app.app_context(). See the documentation for more information.
其中源码是:
resources/celery_task.py
import os # from app import celery # from app import app, log # from flask import current_app as app # from factory import celery from flask import g from resources.tts import refreshAzureSpeechToken app = g.app log = g.log celery = g.celery #---------------------------------------- # Celery tasks #---------------------------------------- # @celery.task() @celery.task # @celery.task(name=app.config["CELERY_TASK_NAME"] + ".deleteTmpAudioFile") def deleteTmpAudioFile(filename): """ delete tmp audio file from filename eg: 98fc7c46-7aa0-4dd7-aa9d-89fdf516abd6.mp3 """ # log = app.logger with celery.app.app_context(): log.info("deleteTmpAudioFile: filename=%s", filename) ...
很明显此处是celery的task中,使用了flask的g,结果此处:
Mac的终端中去运行的celery的worker,没有Flask的app的context了,所以无法正常运行
所以还是要去搞清楚:celery工厂模式,也要支持worker
flask celery factory worker
Celery and the Flask Application Factory Pattern – miguelgrinberg.com
说是:弄个单独的celery_worker.py,然后在里面
from app import celery, create_app
和之前的做法不兼容?
感觉都是单独的创建app和celery的:
from factories.celery import create_celery from factories.application import create_application celery = create_celery(create_application())
这样的话,app岂不是和Flask运行的app不是一个实例了吗?
这个celery也是个app,是个实例,celery application
概念上等价于Flask的app,都是个实例
看似最专业,但是:
也在tasks.py中
from flask import g
此处还是行不通啊
flask celery worker RuntimeError: Working outside of application context
然后代码改回最早的,celery都放到app.py中:
app.py
print("in flask app: settings=%s" % (settings)) app = create_app(settings) app.app_context().push() # register_extensions(app) log = app.logger log.debug("app=%s", app) log.debug("log=%s", log) log.debug("settings.FLASK_ENV=%s", settings.FLASK_ENV) log.debug("settings.DEBUG=%s, settings.MONGODB_HOST=%s, settings.FILE_URL_HOST=%s", settings.DEBUG, settings.MONGODB_HOST, settings.FILE_URL_HOST) # celery = None # with app.app_context(): celery = create_celery_app(app) print("celery=%s" % celery) #---------------------------------------- # Celery tasks #---------------------------------------- # @celery.task() @celery.task # @celery.task(name=app.config["CELERY_TASK_NAME"] + ".deleteTmpAudioFile") def deleteTmpAudioFile(filename): """ delete tmp audio file from filename eg: 98fc7c46-7aa0-4dd7-aa9d-89fdf516abd6.mp3 """ # log = app.logger # with celery.app.app_context(): # log = celery.app.logger # app = celery.app log.info("deleteTmpAudioFile: filename=%s", filename) audioTmpFolder = app.config["AUDIO_TEMP_FOLDER"] # audioTmpFolder = "tmp/audio" log.info("audioTmpFolder=%s", audioTmpFolder) curFolderAbsPath = os.getcwd() #'/Users/crifan/dev/dev_root/company/xxx/projects/robotDemo/server' log.info("curFolderAbsPath=%s", curFolderAbsPath) audioTmpFolderFullPath = os.path.join(curFolderAbsPath, audioTmpFolder) log.info("audioTmpFolderFullPath=%s", audioTmpFolderFullPath) tempAudioFullname = os.path.join(audioTmpFolderFullPath, filename) #'/Users/crifan/dev/dev_root/company/xxx/projects/robotDemo/server/tmp/audio/2aba73d1-f8d0-4302-9dd3-d1dbfad44458.mp3' if os.path.isfile(tempAudioFullname): os.remove(tempAudioFullname) log.info("Ok to delete file %s", tempAudioFullname) else: log.warning("No need to remove for not exist file %s", tempAudioFullname) @celery.task def celeryRefreshAzureSpeechToken(): """celery's task: refreshAzureSpeechToken""" # with celery.app.app_context(): from resources.tts import refreshAzureSpeechToken refreshAzureSpeechToken() @celery.on_after_configure.connect def celerySetupPeriodicTasks(sender, **kwargs): # with celery.app.app_context(): # log = app.logger # log = celery.app.logger # app = celery.app log.info("celerySetupPeriodicTasks: sender=%s", sender) sender.add_periodic_task(app.config["CELERY_REFRESH_MS_TOKEN_INTERVAL"], celeryRefreshAzureSpeechToken.s(), name="refresh ms Azure token every less than 10 minutes")
但是,在别的模块,view中,调用此处的celery的异步函数时:
resources/qa.py
class RobotQaAPI(Resource): def processResponse(self, 。。。 # from resources.tasks import deleteTmpAudioFile from app import deleteTmpAudioFile deleteTmpAudioFile.apply_async([tempFilename], countdown=delayTimeToDelete)
结果导致了app,再去重新初始化的问题。
所以现在问题转换为:
Flask中,在别的模块中,如何导入celery,以及celery中的task函数
flask other view import celery
但是此处好像还是会依赖到别的地方的,包括Flask的app实例-》需要用到其中的log
以及app.config
flask other view call celery task
flask import celery task
去试试:
➜ xxxRobotDemoServer git:(master) ✗ pipenv install Flask-Celery-Helper Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead. Installing Flask-Celery-Helper... Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple Collecting Flask-Celery-Helper Downloading https://pypi.tuna.tsinghua.edu.cn/packages/71/f5/3631b71ac28d9b691ff2e2371a95121be18c742c963098635c11cf4a254f/Flask-Celery-Helper-1.1.0.tar.gz Requirement already satisfied: Flask in /Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages (from Flask-Celery-Helper) (1.0.2) Requirement already satisfied: celery in /Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages (from Flask-Celery-Helper) (4.2.1) Requirement already satisfied: click>=5.1 in /Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages (from Flask->Flask-Celery-Helper) (6.7) Requirement already satisfied: itsdangerous>=0.24 in /Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages (from Flask->Flask-Celery-Helper) (0.24) Requirement already satisfied: Werkzeug>=0.14 in /Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages (from Flask->Flask-Celery-Helper) (0.14.1) Requirement already satisfied: Jinja2>=2.10 in /Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages (from Flask->Flask-Celery-Helper) (2.10) Requirement already satisfied: pytz>dev in /Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages (from celery->Flask-Celery-Helper) (2018.5) Requirement already satisfied: billiard<3.6.0,>=3.5.0.2 in /Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages (from celery->Flask-Celery-Helper) (3.5.0.4) Requirement already satisfied: kombu<5.0,>=4.2.0 in /Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages (from celery->Flask-Celery-Helper) (4.2.1) Requirement already satisfied: MarkupSafe>=0.23 in /Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages (from Jinja2>=2.10->Flask->Flask-Celery-Helper) (1.0) Requirement already satisfied: amqp<3.0,>=2.1.4 in /Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages (from kombu<5.0,>=4.2.0->celery->Flask-Celery-Helper) (2.3.2) Requirement already satisfied: vine>=1.1.3 in /Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages (from amqp<3.0,>=2.1.4->kombu<5.0,>=4.2.0->celery->Flask-Celery-Helper) (1.1.4) Building wheels for collected packages: Flask-Celery-Helper Running setup.py bdist_wheel for Flask-Celery-Helper: started Running setup.py bdist_wheel for Flask-Celery-Helper: finished with status 'done' Stored in directory: /Users/crifan/Library/Caches/pipenv/wheels/de/3a/64/09b65606a3d7523be8e4fecd7fd7b4025086127c633f4fba30 Successfully built Flask-Celery-Helper Installing collected packages: Flask-Celery-Helper Successfully installed Flask-Celery-Helper-1.1.0 Adding Flask-Celery-Helper to Pipfile's [packages]... Pipfile.lock (81f0dc) out of date, updating to (512a06)... Locking [dev-packages] dependencies... Locking [packages] dependencies... Updated Pipfile.lock (512a06)! Installing dependencies from Pipfile.lock (512a06)... 🐍 ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 35/35 — 00:00:14
然后去试试代码
结果还是同样问题:
flask celery worker -A RuntimeError: Working outside of application context
flask celery worker RuntimeError: Working outside of application context
【总结】
最终,想办法去掉了之前celery task中依赖的flask的g,具体做法详见:
【已解决】Flask中如何用工厂模式初始化Celery
转载请注明:在路上 » 【已解决】Flask的Celery改为工厂模式后本地调试worker出错:RuntimeError: Working outside of application context