最新消息:20210816 当前crifan.com域名已被污染,为防止失联,请关注(页面右下角的)公众号

【已解决】Flask的Celery改为工厂模式后本地调试worker出错:RuntimeError: Working outside of application context

Flask crifan 1154浏览 0评论
折腾:
【未解决】用蓝图和工厂模式去优化现有Flask项目代码结构
期间,对于Flask的app的工厂模式的create_app,去初始化时,已经解决了Celery的工厂模式问题:
【已解决】Flask中如何用工厂模式初始化Celery
但是想要完整调试项目,需要Mac的本地终端中去运行celery的worker,结果出错:
➜  xxxRobotDemoServer git:(master) ✗ celery worker -A resources.celery_task.celery --loglevel=DEBUG
Traceback (most recent call last):
  File "/Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/bin/celery", line 11, in <module>
    sys.exit(main())
  File "/Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages/celery/__main__.py", line 16, in main
    _main()
  File "/Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages/celery/bin/celery.py", line 322, in main
    cmd.execute_from_commandline(argv)
  File "/Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages/celery/bin/celery.py", line 496, in execute_from_commandline
    super(CeleryCommand, self).execute_from_commandline(argv)))
  File "/Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages/celery/bin/base.py", line 273, in execute_from_commandline
    argv = self.setup_app_from_commandline(argv)
  File "/Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages/celery/bin/base.py", line 479, in setup_app_from_commandline
    self.app = self.find_app(app)
  File "/Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages/celery/bin/base.py", line 501, in find_app
    return find_app(app, symbol_by_name=self.symbol_by_name)
  File "/Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages/celery/app/utils.py", line 359, in find_app
    sym = symbol_by_name(app, imp=imp)
  File "/Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages/celery/bin/base.py", line 504, in symbol_by_name
    return imports.symbol_by_name(name, imp=imp)
  File "/Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages/kombu/utils/imports.py", line 56, in symbol_by_name
    module = imp(module_name, package=package, **kwargs)
  File "/Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages/celery/utils/imports.py", line 104, in import_from_cwd
    return imp(module, package=package)
  File "/Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/crifan/dev/dev_root/company/xxx/projects/robotDemo/server/xxxRobotDemoServer/resources/celery_task.py", line 7, in <module>
    from resources.tts import refreshAzureSpeechToken
  File "/Users/crifan/dev/dev_root/company/xxx/projects/robotDemo/server/xxxRobotDemoServer/resources/tts.py", line 9, in <module>
    app = g.app
  File "/Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages/werkzeug/local.py", line 347, in __getattr__
    return getattr(self._get_current_object(), name)
  File "/Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages/werkzeug/local.py", line 306, in _get_current_object
    return self.__local()
  File "/Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages/flask/globals.py", line 44, in _lookup_app_object
    raise RuntimeError(_app_ctx_err_msg)
RuntimeError: Working outside of application context.

This typically means that you attempted to use functionality that needed
to interface with the current application object in some way. To solve
this, set up an application context with app.app_context().  See the
documentation for more information.
其中源码是:
resources/celery_task.py
import os
# from app import celery
# from app import app, log
# from flask import current_app as app
# from factory import celery
from flask import g
from resources.tts import refreshAzureSpeechToken

app = g.app
log = g.log
celery = g.celery

#----------------------------------------
# Celery tasks
#----------------------------------------

# @celery.task()
@celery.task
# @celery.task(name=app.config["CELERY_TASK_NAME"] + ".deleteTmpAudioFile")
def deleteTmpAudioFile(filename):
    """
        delete tmp audio file from filename
            eg: 98fc7c46-7aa0-4dd7-aa9d-89fdf516abd6.mp3
    """
    # log = app.logger
    with celery.app.app_context():
        log.info("deleteTmpAudioFile: filename=%s", filename)
        ...
很明显此处是celery的task中,使用了flask的g,结果此处:
Mac的终端中去运行的celery的worker,没有Flask的app的context了,所以无法正常运行
所以还是要去搞清楚:celery工厂模式,也要支持worker
flask celery factory worker
Celery and the Flask Application Factory Pattern – miguelgrinberg.com
https://blog.miguelgrinberg.com/post/celery-and-the-flask-application-factory-pattern
说是:弄个单独的celery_worker.py,然后在里面
from app import celery, create_app
和之前的做法不兼容?
zenyui/celery-flask-factory: Implementing Celery within a Flask application factory
Using Celery with Flask Factories
感觉都是单独的创建app和celery的:
from factories.celery import create_celery
from factories.application import create_application

celery = create_celery(create_application())
这样的话,app岂不是和Flask运行的app不是一个实例了吗?
Celery Background Tasks — Flask 1.0.2 documentation
这个celery也是个app,是个实例,celery application
概念上等价于Flask的app,都是个实例
Flask with create_app, SQLAlchemy and Celery – Stack Overflow
看似最专业,但是:
也在tasks.py中
from flask import g
此处还是行不通啊
flask celery worker RuntimeError: Working outside of application context
Python Flask with celery out of application context – Stack Overflow
python – Celery outside of flask application context – Stack Overflow
python – Flask with Celery – Application context not available – Stack Overflow
然后代码改回最早的,celery都放到app.py中:
app.py
print("in flask app: settings=%s" % (settings))
app = create_app(settings)
app.app_context().push()
# register_extensions(app)
log = app.logger
log.debug("app=%s", app)

log.debug("log=%s", log)
log.debug("settings.FLASK_ENV=%s", settings.FLASK_ENV)
log.debug("settings.DEBUG=%s, settings.MONGODB_HOST=%s, settings.FILE_URL_HOST=%s",
          settings.DEBUG, settings.MONGODB_HOST, settings.FILE_URL_HOST)

# celery = None
# with app.app_context():
celery = create_celery_app(app)
print("celery=%s" % celery)

#----------------------------------------
# Celery tasks
#----------------------------------------

# @celery.task()
@celery.task
# @celery.task(name=app.config["CELERY_TASK_NAME"] + ".deleteTmpAudioFile")
def deleteTmpAudioFile(filename):
    """
        delete tmp audio file from filename
            eg: 98fc7c46-7aa0-4dd7-aa9d-89fdf516abd6.mp3
    """
    # log = app.logger
    # with celery.app.app_context():
    #     log = celery.app.logger
    #     app = celery.app
    log.info("deleteTmpAudioFile: filename=%s", filename)

    audioTmpFolder = app.config["AUDIO_TEMP_FOLDER"]
    # audioTmpFolder = "tmp/audio"
    log.info("audioTmpFolder=%s", audioTmpFolder)
    curFolderAbsPath = os.getcwd() #'/Users/crifan/dev/dev_root/company/xxx/projects/robotDemo/server'
    log.info("curFolderAbsPath=%s", curFolderAbsPath)
    audioTmpFolderFullPath = os.path.join(curFolderAbsPath, audioTmpFolder)
    log.info("audioTmpFolderFullPath=%s", audioTmpFolderFullPath)
    tempAudioFullname = os.path.join(audioTmpFolderFullPath, filename)
    #'/Users/crifan/dev/dev_root/company/xxx/projects/robotDemo/server/tmp/audio/2aba73d1-f8d0-4302-9dd3-d1dbfad44458.mp3'
    if os.path.isfile(tempAudioFullname):
        os.remove(tempAudioFullname)
        log.info("Ok to delete file %s", tempAudioFullname)
    else:
        log.warning("No need to remove for not exist file %s", tempAudioFullname)


@celery.task
def celeryRefreshAzureSpeechToken():
    """celery's task: refreshAzureSpeechToken"""
    # with celery.app.app_context():
    from resources.tts import refreshAzureSpeechToken
    refreshAzureSpeechToken()

@celery.on_after_configure.connect
def celerySetupPeriodicTasks(sender, **kwargs):
    # with celery.app.app_context():
    # log = app.logger
    # log = celery.app.logger
    # app = celery.app
    log.info("celerySetupPeriodicTasks: sender=%s", sender)

    sender.add_periodic_task(app.config["CELERY_REFRESH_MS_TOKEN_INTERVAL"],
                             celeryRefreshAzureSpeechToken.s(),
                             name="refresh ms Azure token every less than 10 minutes")
但是,在别的模块,view中,调用此处的celery的异步函数时:
resources/qa.py
class RobotQaAPI(Resource):

    def processResponse(self,
        。。。
        
# from resources.tasks import deleteTmpAudioFile
        from app import deleteTmpAudioFile
        deleteTmpAudioFile.apply_async([tempFilename], countdown=delayTimeToDelete)
结果导致了app,再去重新初始化的问题。
所以现在问题转换为:
Flask中,在别的模块中,如何导入celery,以及celery中的task函数
flask other view import celery
Flask integration with Celery – Stack Overflow
python – ImportError: cannot import name celery – Stack Overflow
First steps with Django — Celery 4.2.0 documentation
但是此处好像还是会依赖到别的地方的,包括Flask的app实例-》需要用到其中的log
以及app.config
flask-appfactory/examples/myapp at master · inveniosoftware-attic/flask-appfactory
Flask, blueprints uses celery task and got cycle import – Stack Overflow
flask other view call celery task
Celery tasks are always pending (Python Flask) – Stack Overflow
python – Flask celery tasks not working – Stack Overflow
flask import celery task
去试试:
Robpol86/Flask-Celery-Helper: Celery support for Flask without breaking PyCharm inspections.
➜  xxxRobotDemoServer git:(master) ✗ pipenv install Flask-Celery-Helper
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead.
Installing Flask-Celery-Helper...
Looking in indexes:
 
https://pypi.tuna.tsinghua.edu.cn/simple
Collecting Flask-Celery-Helper
  Downloading
 
https://pypi.tuna.tsinghua.edu.cn/packages/71/f5/3631b71ac28d9b691ff2e2371a95121be18c742c963098635c11cf4a254f/Flask-Celery-Helper-1.1.0.tar.gz
Requirement already satisfied: Flask in /Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages (from Flask-Celery-Helper) (1.0.2)
Requirement already satisfied: celery in /Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages (from Flask-Celery-Helper) (4.2.1)
Requirement already satisfied: click>=5.1 in /Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages (from Flask->Flask-Celery-Helper) (6.7)
Requirement already satisfied: itsdangerous>=0.24 in /Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages (from Flask->Flask-Celery-Helper) (0.24)
Requirement already satisfied: Werkzeug>=0.14 in /Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages (from Flask->Flask-Celery-Helper) (0.14.1)
Requirement already satisfied: Jinja2>=2.10 in /Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages (from Flask->Flask-Celery-Helper) (2.10)
Requirement already satisfied: pytz>dev in /Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages (from celery->Flask-Celery-Helper) (2018.5)
Requirement already satisfied: billiard<3.6.0,>=3.5.0.2 in /Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages (from celery->Flask-Celery-Helper) (3.5.0.4)
Requirement already satisfied: kombu<5.0,>=4.2.0 in /Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages (from celery->Flask-Celery-Helper) (4.2.1)
Requirement already satisfied: MarkupSafe>=0.23 in /Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages (from Jinja2>=2.10->Flask->Flask-Celery-Helper) (1.0)
Requirement already satisfied: amqp<3.0,>=2.1.4 in /Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages (from kombu<5.0,>=4.2.0->celery->Flask-Celery-Helper) (2.3.2)
Requirement already satisfied: vine>=1.1.3 in /Users/crifan/.local/share/virtualenvs/xxxRobotDemoServer-SCpLPEyZ/lib/python3.6/site-packages (from amqp<3.0,>=2.1.4->kombu<5.0,>=4.2.0->celery->Flask-Celery-Helper) (1.1.4)
Building wheels for collected packages: Flask-Celery-Helper
  Running setup.py bdist_wheel for Flask-Celery-Helper: started
  Running setup.py bdist_wheel for Flask-Celery-Helper: finished with status 'done'
  Stored in directory: /Users/crifan/Library/Caches/pipenv/wheels/de/3a/64/09b65606a3d7523be8e4fecd7fd7b4025086127c633f4fba30
Successfully built Flask-Celery-Helper
Installing collected packages: Flask-Celery-Helper
Successfully installed Flask-Celery-Helper-1.1.0

Adding Flask-Celery-Helper to Pipfile's [packages]...
Pipfile.lock (81f0dc) out of date, updating to (512a06)...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Updated Pipfile.lock (512a06)!
Installing dependencies from Pipfile.lock (512a06)...
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 35/35 — 00:00:14
然后去试试代码
结果还是同样问题:
flask celery worker -A RuntimeError: Working outside of application context
flask celery worker  RuntimeError: Working outside of application context
【总结】
最终,想办法去掉了之前celery task中依赖的flask的g,具体做法详见:
【已解决】Flask中如何用工厂模式初始化Celery

转载请注明:在路上 » 【已解决】Flask的Celery改为工厂模式后本地调试worker出错:RuntimeError: Working outside of application context

发表我的评论
取消评论

表情

Hi,您需要填写昵称和邮箱!

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址
84 queries in 0.166 seconds, using 22.08MB memory