最新消息:20210816 当前crifan.com域名已被污染,为防止失联,请关注(页面右下角的)公众号

【已解决】用gunicorn的gevent解决之前多worker多Process线程的单例的数据共享

gunicorn crifan 7862浏览 0评论

折腾:

【已解决】Flask的gunicorn中多进程多worker如何共享数据或单实例

期间,暂时不想去试试多线程的其他共享内存,mmap等复杂的方法。

而先去试试这个方法:gunicorn的gevent

即:gunicorn中,把之前多worker,多process线程,

改为单worker,worker的type

看看能否解决问题。

先去试试:gunicorn的gevent

How to get a concurrency of 1000 requests with Flask and Gunicorn – Stack Overflow

flask gunicorn gevent coroutine

django flask应用uwsgi和tornado异步回调在项目中的体验 | 峰云就她了

python – How many concurrent requests does a single Flask process receive? – Stack Overflow

Design — Gunicorn 19.9.0 documentation

Introduction — gevent 1.3.6.dev0 documentation

Gunicorn Workers and Threads – Stack Overflow

http://docs.gunicorn.org/en/stable/settings.html#worker-class

先去安装gevent:

<code>➜  xxx git:(master) ✗ pipenv install gevent
Installing gevent...
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Collecting gevent
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/2b/a9/7c38605b9672a6ede6ccf822a645fdeec0c80fb467c87c5ce4976e4056dd/gevent-1.3.6-cp36-cp36m-macosx_10_6_intel.whl (3.1MB)
Collecting greenlet&gt;=0.4.14; platform_python_implementation == "CPython" (from gevent)
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/5d/82/2e53a8def6f99db51992ca3a0a2448c3bbec1a9db3a7cbf7d5dad011e138/greenlet-0.4.14.tar.gz (59kB)
Building wheels for collected packages: greenlet
  Running setup.py bdist_wheel for greenlet: started
  Running setup.py bdist_wheel for greenlet: finished with status 'done'
  Stored in directory: /Users/crifan/Library/Caches/pipenv/wheels/3f/20/d5/59a7f616d60eb238802e9860811aa4a069307c8fba12fce6d6
Successfully built greenlet
Installing collected packages: greenlet, gevent
Successfully installed gevent-1.3.6 greenlet-0.4.14

Adding gevent to Pipfile's [packages]...
Pipfile.lock not found, creating...
Locking [dev-packages] dependencies...
</code>

django flask应用uwsgi和tornado异步回调在项目中的体验 | 峰云就她了

截止到 2018 年 1 月,Python 的 asyncio 的生态如何? – 知乎

所以现在是:

http://docs.gunicorn.org/en/stable/settings.html#worker-class

设置为gevent了,然后:

http://docs.gunicorn.org/en/stable/settings.html#workers

workers设置为1

而对于

threads

http://docs.gunicorn.org/en/stable/settings.html#threads

worker_connections

http://docs.gunicorn.org/en/stable/settings.html#worker-connections

max_requests

http://docs.gunicorn.org/en/stable/settings.html#max-requests

到底设置为多少,去参考:

Gunicorn Workers and Threads – Stack Overflow

目前结论:

  • threads:无需设置

    • 官网说了:此选项只适用于Gthread类型的worker

  • worker_connections:此处随便设置,默认本身是1000

    • 注:只适用于Eventlet和Gevent类型的worker

  • max_requests:默认是0,表示不限制,除非想要降低内存泄漏风险,否则可以不设置

但是CentOS中去用pipenv安装gevent失败:

<code>[root@xxx-general-01 robotDemo]# pipenv install
Pipfile.lock (ab35a7) out of date, updating to (feef41)…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (feef41)!
Installing dependencies from Pipfile.lock (feef41)…
Ignoring greenlet: markers 'platform_python_implementation == "cpython"' don't match your environment
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 62/62 — 00:00:13
To activate this project's virtualenv, run the following:
 $ pipenv shell
</code>

不过通过graph发现已安装:

<code>[root@xxx-general-01 robotDemo]# pipenv graph
celery==4.2.1
  - billiard [required: &gt;=3.5.0.2,&lt;3.6.0, installed: 3.5.0.4]
  - kombu [required: &gt;=4.2.0,&lt;5.0, installed: 4.2.1]
    - amqp [required: &gt;=2.1.4,&lt;3.0, installed: 2.3.2]
      - vine [required: &gt;=1.1.3, installed: 1.1.4]
  - pytz [required: &gt;dev, installed: 2018.5]
Flask-Cors==3.0.6
  - Flask [required: &gt;=0.9, installed: 1.0.2]
    - click [required: &gt;=5.1, installed: 6.7]
    - itsdangerous [required: &gt;=0.24, installed: 0.24]
    - Jinja2 [required: &gt;=2.10, installed: 2.10]
      - MarkupSafe [required: &gt;=0.23, installed: 1.0]
    - Werkzeug [required: &gt;=0.14, installed: 0.14.1]
  - Six [required: Any, installed: 1.11.0]
Flask-PyMongo==2.1.0
  - Flask [required: &gt;=0.11, installed: 1.0.2]
    - click [required: &gt;=5.1, installed: 6.7]
    - itsdangerous [required: &gt;=0.24, installed: 0.24]
    - Jinja2 [required: &gt;=2.10, installed: 2.10]
      - MarkupSafe [required: &gt;=0.23, installed: 1.0]
    - Werkzeug [required: &gt;=0.14, installed: 0.14.1]
  - PyMongo [required: &gt;=3.0, installed: 3.7.1]
Flask-RESTful==0.3.6
  - aniso8601 [required: &gt;=0.82, installed: 3.0.2]
  - Flask [required: &gt;=0.8, installed: 1.0.2]
    - click [required: &gt;=5.1, installed: 6.7]
    - itsdangerous [required: &gt;=0.24, installed: 0.24]
    - Jinja2 [required: &gt;=2.10, installed: 2.10]
      - MarkupSafe [required: &gt;=0.23, installed: 1.0]
    - Werkzeug [required: &gt;=0.14, installed: 0.14.1]
  - pytz [required: Any, installed: 2018.5]
  - six [required: &gt;=1.3.0, installed: 1.11.0]
gensim==3.5.0
  - numpy [required: &gt;=1.11.3, installed: 1.15.1]
  - scipy [required: &gt;=0.18.1, installed: 1.1.0]
    - numpy [required: &gt;=1.8.2, installed: 1.15.1]
  - six [required: &gt;=1.5.0, installed: 1.11.0]
  - smart-open [required: &gt;=1.2.1, installed: 1.6.0]
    - boto [required: &gt;=2.32, installed: 2.49.0]
    - boto3 [required: Any, installed: 1.8.3]
      - botocore [required: &lt;1.12.0,&gt;=1.11.3, installed: 1.11.3]
        - docutils [required: &gt;=0.10, installed: 0.14]
        - jmespath [required: &lt;1.0.0,&gt;=0.7.1, installed: 0.9.3]
        - python-dateutil [required: &gt;=2.1,&lt;3.0.0, installed: 2.7.3]
          - six [required: &gt;=1.5, installed: 1.11.0]
        - urllib3 [required: &lt;1.24,&gt;=1.20, installed: 1.23]
      - jmespath [required: &lt;1.0.0,&gt;=0.7.1, installed: 0.9.3]
      - s3transfer [required: &lt;0.2.0,&gt;=0.1.10, installed: 0.1.13]
        - botocore [required: &gt;=1.3.0,&lt;2.0.0, installed: 1.11.3]
          - docutils [required: &gt;=0.10, installed: 0.14]
          - jmespath [required: &lt;1.0.0,&gt;=0.7.1, installed: 0.9.3]
          - python-dateutil [required: &gt;=2.1,&lt;3.0.0, installed: 2.7.3]
            - six [required: &gt;=1.5, installed: 1.11.0]
          - urllib3 [required: &lt;1.24,&gt;=1.20, installed: 1.23]
    - bz2file [required: Any, installed: 0.98]
    - requests [required: Any, installed: 2.19.1]
      - certifi [required: &gt;=2017.4.17, installed: 2018.8.24]
      - chardet [required: &lt;3.1.0,&gt;=3.0.2, installed: 3.0.4]
      - idna [required: &lt;2.8,&gt;=2.5, installed: 2.7]
      - urllib3 [required: &lt;1.24,&gt;=1.21.1, installed: 1.23]
gevent==1.3.6
  - greenlet [required: &gt;=0.4.14, installed: ?]
gunicorn==19.9.0
jieba==0.39
openpyxl==2.5.5
  - et-xmlfile [required: Any, installed: 1.0.1]
  - jdcal [required: Any, installed: 1.4]
patchwork==1.0.1
  - fabric [required: &gt;=2.0,&lt;3.0, installed: 2.3.1]
    - cryptography [required: &gt;=1.1, installed: 2.3.1]
      - asn1crypto [required: &gt;=0.21.0, installed: 0.24.0]
      - cffi [required: &gt;=1.7,!=1.11.3, installed: 1.11.5]
        - pycparser [required: Any, installed: 2.18]
      - idna [required: &gt;=2.1, installed: 2.7]
      - six [required: &gt;=1.4.1, installed: 1.11.0]
    - invoke [required: &lt;2.0,&gt;=1.1, installed: 1.1.1]
    - paramiko [required: &gt;=2.4, installed: 2.4.1]
      - bcrypt [required: &gt;=3.1.3, installed: 3.1.4]
        - cffi [required: &gt;=1.1, installed: 1.11.5]
          - pycparser [required: Any, installed: 2.18]
        - six [required: &gt;=1.4.1, installed: 1.11.0]
      - cryptography [required: &gt;=1.5, installed: 2.3.1]
        - asn1crypto [required: &gt;=0.21.0, installed: 0.24.0]
        - cffi [required: &gt;=1.7,!=1.11.3, installed: 1.11.5]
          - pycparser [required: Any, installed: 2.18]
        - idna [required: &gt;=2.1, installed: 2.7]
        - six [required: &gt;=1.4.1, installed: 1.11.0]
      - pyasn1 [required: &gt;=0.1.7, installed: 0.4.4]
      - pynacl [required: &gt;=1.0.1, installed: 1.2.1]
        - cffi [required: &gt;=1.4.1, installed: 1.11.5]
          - pycparser [required: Any, installed: 2.18]
        - six [required: Any, installed: 1.11.0]
pycnnum==1.0.1
pylru==1.1.0
PyMySQL==0.9.2
  - cryptography [required: Any, installed: 2.3.1]
    - asn1crypto [required: &gt;=0.21.0, installed: 0.24.0]
    - cffi [required: &gt;=1.7,!=1.11.3, installed: 1.11.5]
      - pycparser [required: Any, installed: 2.18]
    - idna [required: &gt;=2.1, installed: 2.7]
    - six [required: &gt;=1.4.1, installed: 1.11.0]
python-dotenv==0.9.1
redis==2.10.6
schedule==0.5.0
SolrClient==0.2.1
  - kazoo [required: ==2.2.1, installed: 2.2.1]
    - six [required: Any, installed: 1.11.0]
  - requests [required: &gt;=2.2.1, installed: 2.19.1]
    - certifi [required: &gt;=2017.4.17, installed: 2018.8.24]
    - chardet [required: &lt;3.1.0,&gt;=3.0.2, installed: 3.0.4]
    - idna [required: &lt;2.8,&gt;=2.5, installed: 2.7]
    - urllib3 [required: &lt;1.24,&gt;=1.21.1, installed: 1.23]
tqdm==4.25.0
</code>

然后去看看效果

发现log提示缺少greenlet,所以去本地和在线都安装:

<code>[root@xxx-general-01 logs]# pipenv install greenlet
Installing greenlet…
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Collecting greenlet
  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/de/7b/cb662640540725deb0627264f6b890ee2b7725848b8cbca49e27bf3273c6/greenlet-0.4.14-cp36-cp36m-manylinux1_x86_64.whl
Installing collected packages: greenlet
Successfully installed greenlet-0.4.14

Adding greenlet to Pipfile's [packages]…
Pipfile.lock (feef41) out of date, updating to (36cd02)…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (36cd02)!
Installing dependencies from Pipfile.lock (36cd02)…
   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 62/62 — 00:00:12
To activate this project's virtualenv, run the following:
 $ pipenv shell
</code>

安装后,Pipfile中就有了:

<code>gevent = "*"
greenlet = "*"
</code>

然后再去 运行试试

为了更好的调试,发现时不时多个不同线程调用单例的初始化的,所以再去给logging中加上process信息:

参考:

https://docs.python.org/2/library/logging.html#logrecord-attributes

加上:

process
%(process)d
Process ID (if available).
processName
%(processName)s
Process name (if available).

变成:

<code>LOG_FORMAT = "[%(asctime)s %(levelname)s %(process)d %(processName)s %(filename)s:%(lineno)d %(funcName)s] %(message)s"
</code>

然后再去看看log

然后看到对应的log日志输出是:

logs/gunicorn_error.log

<code>[2018-08-30 13:28:35 +0800] [26048] [INFO] Starting gunicorn 19.9.0
[2018-08-30 13:28:35 +0800] [26048] [DEBUG] Arbiter booted
[2018-08-30 13:28:35 +0800] [26048] [INFO] Listening at: http://0.0.0.0:32851 (26048)
[2018-08-30 13:28:35 +0800] [26048] [INFO] Using worker: gevent
[2018-08-30 13:28:35 +0800] [26063] [INFO] Booting worker with pid: 26063
[2018-08-30 13:28:35 +0800] [26048] [DEBUG] 1 workers
</code>

是所希望的

单个worker

type是gevent

然后对应的所希望的单例:

gMsTtsTokenSingleton=

对于Flask涉及的gunicorn中woker中process,的确也是单个实例了。

【总结】

此处,是可以通过gunicorn部署Flask时,把

conf/gunicorn/gunicorn_config.py

<code>worker_class = 'sync'                           #默认的是sync模式
workers = multiprocessing.cpu_count() * 2 + 1   #进程数
</code>

改为:

<code># http://docs.gunicorn.org/en/stable/settings.html#worker-class
worker_class = 'gevent'
workers = 1   #进程数
# http://docs.gunicorn.org/en/stable/settings.html#max-requests
worker_connections = 2000 # 默认是1000
# http://docs.gunicorn.org/en/stable/settings.html#max-requests
max_requests = 0
</code>

即:

从woker=9,type=sync改为worker=1,type=gevent

从而使得:确保Flask中的app,从9个Process,变成单个Process了。

从而单个Process中的Flask的APP的单例,就可以正常工作了。

只不过又出现其他方面的问题:

【未解决】Flask中gunicorn部署和supervisor管理celery的worker导致多线程导致单例失效

转载请注明:在路上 » 【已解决】用gunicorn的gevent解决之前多worker多Process线程的单例的数据共享

发表我的评论
取消评论

表情

Hi,您需要填写昵称和邮箱!

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址
82 queries in 0.193 seconds, using 22.17MB memory