最新消息:20210816 当前crifan.com域名已被污染,为防止失联,请关注(页面右下角的)公众号

【记录】把合并了基于搜索的兜底对话的产品demo部署到在线环境中

工作和技术 crifan 2389浏览 0评论

折腾:

【已解决】合并基于搜索的兜底对话到产品Demo中

期间,已经本地调试完毕了,现在需要上传并部署到线上环境:

其中:

去 删除之前的虚拟环境,停止之前的服务:

<code>(robotDemo-dwdcgdaG) [root@xxx-general-01 robotDemo]# pipenv --rm
Removing virtualenv (/root/.local/share/virtualenvs/robotDemo-dwdcgdaG)…
(robotDemo-dwdcgdaG) [root@xxx-general-01 robotDemo]# ll /root/.local/share/virtualenvs/
total 12
drwxr-xr-x 5 root root 4096 Jul 17 17:16 BatchImportScript-vVMBoBec
drwxr-xr-x 5 root root 4096 Jun  6 17:45 debug_nlp-wDGPvi3o
drwxr-xr-x 5 root root 4096 Jun  7 13:53 xxx-VJ297lRu
(robotDemo-dwdcgdaG) [root@xxx-general-01 robotDemo]# ll /root/.virtualenvs/
total 12
drwxr-xr-x 5 root root 4096 Jul 17 17:16 BatchImportScript-vVMBoBec
drwxr-xr-x 5 root root 4096 Jun  6 17:45 debug_nlp-wDGPvi3o
drwxr-xr-x 5 root root 4096 Jun  7 13:53 xxx-VJ297lRu
(robotDemo-dwdcgdaG) [root@xxx-general-01 robotDemo]# pwd
/root/xxx/server/robotDemo
(robotDemo-dwdcgdaG) [root@xxx-general-01 robotDemo]# ll
total 104
drwxr-xr-x 5 root root  4096 May 10 14:57 ai
-rw-r--r-- 1 root root 41549 Jul 25 11:12 app.py
-rw-r--r-- 1 root root     6 Jun 25 11:04 celerybeat.pid
-rw-r--r-- 1 root root  2252 Jul  3 17:13 config.py
-rw-r--r-- 1 root root  2005 Apr 24 16:36 gunicorn_config.py
drwxr-xr-x 2 root root  4096 Aug 25 06:11 logs
-rw-r--r-- 1 root root   303 May 11 17:22 Pipfile
-rw-r--r-- 1 root root 12454 May 11 17:24 Pipfile.lock
drwxr-xr-x 2 root root  4096 Jul 25 11:19 __pycache__
-rw-r--r-- 1 root root  2102 May  2 16:10 README.md
drwxr-xr-x 2 root root  4096 Aug 27 17:14 runtime
-rw-r--r-- 1 root root  2447 May 25 17:34 supervisord_server.conf
drwxr-xr-x 3 root root  4096 May 14 16:17 tmp
(robotDemo-dwdcgdaG) [root@xxx-general-01 robotDemo]# pwd
/root/xxx/server/robotDemo
(robotDemo-dwdcgdaG) [root@xxx-general-01 robotDemo]# supervisorctl status
gunicorn                         RUNNING   pid 16426, uptime 16 days, 3:57:45
redis                            RUNNING   pid 21685, uptime 63 days, 6:20:12
robotDemo                        RUNNING   pid 19658, uptime 33 days, 6:05:03
robotDemo_CeleryBeat             RUNNING   pid 21684, uptime 63 days, 6:20:12
robotDemo_CeleryWorker           RUNNING   pid 21687, uptime 63 days, 6:20:12
(robotDemo-dwdcgdaG) [root@xxx-general-01 robotDemo]# supervisorctl stop robotDemo robotDemo_CeleryBeat robotDemo_CeleryWorker redis
robotDemo: stopped
robotDemo_CeleryBeat: stopped
robotDemo_CeleryWorker: stopped
redis: stopped
(robotDemo-dwdcgdaG) [root@xxx-general-01 robotDemo]# supervisorctl status
gunicorn                         RUNNING   pid 16426, uptime 16 days, 3:58:37
redis                            STOPPED   Aug 27 05:24 PM
robotDemo                        STOPPED   Aug 27 05:24 PM
robotDemo_CeleryBeat             STOPPED   Aug 27 05:24 PM
robotDemo_CeleryWorker           STOPPED   Aug 27 05:24 PM
(robotDemo-dwdcgdaG) [root@xxx-general-01 robotDemo]#
</code>

然后把之前代码改名留后续删除:

<code>(robotDemo-dwdcgdaG) [root@xxx-general-01 server]# ll -lha
total 20K
drwxr-xr-x 5 root root 4.0K Aug 11 13:26 .
drwxr-xr-x 5 root root 4.0K Jul 17 17:14 ..
drwxr-xr-x 2 root root 4.0K Jul 25 09:53 for_bacup_mysql
drwxr-xr-x 8 root root 4.0K Jul 31 11:36 xxxCmsServer
srwxrwxrwx 1 root root    0 Aug 11 13:26 xxxCmsServer.sock
drwxr-xr-x 7 root root 4.0K Aug 27 17:24 robotDemo
(robotDemo-dwdcgdaG) [root@xxx-general-01 server]# mv robotDemo/ robotDemo_toDelete
(robotDemo-dwdcgdaG) [root@xxx-general-01 server]# ll -lha
total 20K
drwxr-xr-x 5 root root 4.0K Aug 27 17:57 .
drwxr-xr-x 5 root root 4.0K Jul 17 17:14 ..
drwxr-xr-x 2 root root 4.0K Jul 25 09:53 for_bacup_mysql
drwxr-xr-x 8 root root 4.0K Jul 31 11:36 xxxCmsServer
srwxrwxrwx 1 root root    0 Aug 11 13:26 xxxCmsServer.sock
drwxr-xr-x 7 root root 4.0K Aug 27 17:24 robotDemo_toDelete
(robotDemo-dwdcgdaG) [root@xxx-general-01 server]# mkdir robotDemo
(robotDemo-dwdcgdaG) [root@xxx-general-01 server]# ll -lha
total 24K
drwxr-xr-x 6 root root 4.0K Aug 27 17:58 .
drwxr-xr-x 5 root root 4.0K Jul 17 17:14 ..
drwxr-xr-x 2 root root 4.0K Jul 25 09:53 for_bacup_mysql
drwxr-xr-x 8 root root 4.0K Jul 31 11:36 xxxCmsServer
srwxrwxrwx 1 root root    0 Aug 11 13:26 xxxCmsServer.sock
drwxr-xr-x 2 root root 4.0K Aug 27 17:58 robotDemo
drwxr-xr-x 7 root root 4.0K Aug 27 17:24 robotDemo_toDelete
(robotDemo-dwdcgdaG) [root@xxx-general-01 server]# pwd
/root/xxx/web/server
</code>

然后重新用Fabric:

/Users/crifan/dev/dev_root/xxx/deploy/fabfile.py

<code>from invoke import task
from fabric import Connection
from patchwork.transfers import rsync
import os

RemoteHost = '47.x.x.x'
RemoteUser = 'xxx'
RemotePathRoot = '/root/xxx/robotDemo'

def seperatorLine(seperatorChar='-', count=80):
    print(seperatorChar * count)

def fabLoadedPath():
    from fabric.main import program
    return program.collection.loaded_from

@task
def upload(context):
    print("upload: context=%s", context)
    seperatorLine()
    fabFilePath = fabLoadedPath()
    print("fabFilePath=%s" % fabFilePath)
    localProjectRootPath = os.path.join(fabFilePath, "..")
    print("localProjectRootPath=%s" % localProjectRootPath)
    seperatorLine()
    print("Local environment:")
    context.run("uname -a")
    context.run("pwd")
    context.run("ls -lha")
    seperatorLine()
    remoteConn = Connection(host=RemoteHost, user=RemoteUser)
    print(remoteConn)
    seperatorLine()
    print("Remote Server:")
    remoteConn.run('uname -a')
    remoteConn.run('pwd')
    print("remote path: %s" % RemotePathRoot)
    remoteConn.run('ls -lha %s' % (RemotePathRoot))
    # remoteConn.run('cd %s &amp;&amp; pwd &amp;&amp; ls -lha' % RemotePathRoot)
    # putFileResult = remoteConn.put('fabfile.py', remote=RemotePathRoot)
    # print("Uploaded {0.local} to {0.remote}".format(putFileResult))

    # syncSource = "../deploy"
    syncSource = localProjectRootPath
    syncTarget = RemotePathRoot
    # syncExclude = [".DS_Store", "data/", "processData/", "*.log", "*.pyc", "__pycache__"]
    syncExclude = [
        ".DS_Store", ".git/", ".idea/", "*.pyc", "__pycache__",
        "dump.rdb", "debug/", "logs/", "runtime/", "tmp/"]
    syncResp = rsync(
        remoteConn,
        source=syncSource,
        target=syncTarget,
        # Note: be careful to add `delete`, to void to delete unexpected files
        # delete=True,
        exclude=syncExclude)
    print("Sync lcoal %s to remote %s while exclude %s -&gt; return %s" %
          (syncSource, syncTarget, syncExclude, syncResp))
</code>

上传本地代码:

<code>Sync lcoal /Users/xxx/deploy/.. to remote /root/x/web/server/robotDemo while exclude ['.DS_Store', '.git/', '.idea/', '*.pyc', '__pycache__', 'dump.rdb', 'debug/', 'logs/', 'runtime/', 'tmp/'] -&gt; return Command exited with status 0.
=== stdout ===
building file list ... done
./
celerybeat.pid
deploy/
deploy/fabfile.py

sent 1.29K bytes  received 100 bytes  926.00 bytes/sec
total size is 77.13K  speedup is 55.53

(no stderr)
➜  xx git:(master) ✗ fab -r deploy upload
</code>

在线就可以看到代码了,并且也新建了对应的几个原本忽略掉的文件夹:

<code>(robotDemo-dwdcgdaG) [root@x-general-01 robotDemo]# pwd
/root/x/web/server/robotDemo
(robotDemo-dwdcgdaG) [root@x-general-01 robotDemo]# mkdir logs
(robotDemo-dwdcgdaG) [root@x-general-01 robotDemo]# mkdir runtime
(robotDemo-dwdcgdaG) [root@x-general-01 robotDemo]# mkdir tmp
(robotDemo-dwdcgdaG) [root@x-general-01 robotDemo]# ll -lha
total 68K
drwxr-xr-x 9 root root 4.0K Aug 27 14:19 .
drwxr-xr-x 6 root root 4.0K Aug 27 17:58 ..
-rw-r--r-- 1 root root 1.5K Aug 27 10:26 app.py
-rw-r--r-- 1 root root    6 Aug 27 14:19 celerybeat.pid
drwxr-xr-x 2 root root 4.0K Aug 27 17:19 common
drwxr-xr-x 5 root root 4.0K Aug 22 14:50 conf
drwxr-xr-x 2 root root 4.0K Aug 27 18:03 deploy
-rw-r--r-- 1 root root 6.6K Aug 25 15:07 factory.py
-rw-r--r-- 1 root root  162 May 24 11:52 .gitignore
drwxr-xr-x 2 root root 4.0K Aug 27 18:02 logs
-rw-r--r-- 1 root root  495 Aug 27 10:35 Pipfile
-rw-r--r-- 1 root root 6.2K Aug 27 14:03 README.md
drwxr-xr-x 2 root root 4.0K Aug 27 17:05 resources
drwxr-xr-x 2 root root 4.0K Aug 27 18:02 runtime
drwxr-xr-x 2 root root 4.0K Aug 27 18:02 tmp
(robotDemo-dwdcgdaG) [root@xx-general-01 robotDemo]#
</code>

重新创建pipenv的虚拟环境:

<code>(robotDemo-dwdcgdaG) [root@x-general-01 robotDemo]# pipenv --rm
No virtualenv has been created for this project yet!
(robotDemo-dwdcgdaG) [root@x-general-01 robotDemo]# rm /root/.local/share/virtualenvs/
BatchImportScript-vVMBoBec/ debug_nlp-wDGPvi3o/         x-VJ297lRu/         
(robotDemo-dwdcgdaG) [root@x-general-01 robotDemo]# rm /root/.local/share/virtualenvs/^C
(robotDemo-dwdcgdaG) [root@x-general-01 robotDemo]# ll /root/.virtualenvs/
BatchImportScript-vVMBoBec/ debug_nlp-wDGPvi3o/         x-VJ297lRu/         
(robotDemo-dwdcgdaG) [root@x-general-01 robotDemo]# cat Pipfile
[[source]]
#url = "https://pypi.python.org/simple"
url = "https://pypi.tuna.tsinghua.edu.cn/simple"
verify_ssl = true
name = "pypi"

[packages]
flask = "*"
flask-restful = "*"
flask-pymongo = "*"
gunicorn = "*"
pymysql = "*"
openpyxl = "*"
flask-cors = "*"
requests = "*"
celery = {extras = ["redis"]}
python-dotenv = "*"
numpy = "*"
schedule = "*"
pylru = "*"
jieba = "*"
pycnnum = "*"
solrclient = "*"
gensim = "*"
patchwork = "*"
tqdm = "*"

[dev-packages]

[requires]
python_version = "3.6"
(robotDemo-dwdcgdaG) [root@x-general-01 robotDemo]# pipenv install
Creating a virtualenv for this project…
Using /root/miniconda3/bin/python3.6m (3.6.5) to create virtualenv…
⠋Running virtualenv with interpreter /root/miniconda3/bin/python3.6m
Using base prefix '/root/miniconda3'
New python executable in /root/.local/share/virtualenvs/robotDemo-dwdcgdaG/bin/python3.6m
Also creating executable in /root/.local/share/virtualenvs/robotDemo-dwdcgdaG/bin/python
Please make sure you remove any previous custom paths from your /root/.pydistutils.cfg file.
Installing setuptools, pip, wheel...done.

Virtualenv location: /root/.local/share/virtualenvs/robotDemo-dwdcgdaG
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…

...
</code>

感觉死掉了:

又卡死了。

重新试试:

<code>(robotDemo-dwdcgdaG) [root@xx-general-01 robotDemo]# pipenv --rm
Removing virtualenv (/root/.local/share/virtualenvs/robotDemo-dwdcgdaG)…
(robotDemo-dwdcgdaG) [root@xx-general-01 robotDemo]# pipenv install
...
</code>

终于开始安装库了:

<code>(robotDemo-dwdcgdaG) [root@xx-general-01 robotDemo]# pipenv install
Creating a virtualenv for this project…
Using /root/miniconda3/bin/python3.6m (3.6.5) to create virtualenv…
⠋Running virtualenv with interpreter /root/miniconda3/bin/python3.6m
Using base prefix '/root/miniconda3'
New python executable in /root/.local/share/virtualenvs/robotDemo-dwdcgdaG/bin/python3.6m
Also creating executable in /root/.local/share/virtualenvs/robotDemo-dwdcgdaG/bin/python
Please make sure you remove any previous custom paths from your /root/.pydistutils.cfg file.
Installing setuptools, pip, wheel...done.

Virtualenv location: /root/.local/share/virtualenvs/robotDemo-dwdcgdaG
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (ab35a7)!
Installing dependencies from Pipfile.lock (ab35a7)…
   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 60/60 — 00:00:28
(robotDemo-dwdcgdaG) [root@x-general-01 robotDemo]# pipenv graph
celery==4.2.1
  - billiard [required: &gt;=3.5.0.2,&lt;3.6.0, installed: 3.5.0.4]
  - kombu [required: &lt;5.0,&gt;=4.2.0, installed: 4.2.1]
    - amqp [required: &lt;3.0,&gt;=2.1.4, installed: 2.3.2]
      - vine [required: &gt;=1.1.3, installed: 1.1.4]
  - pytz [required: &gt;dev, installed: 2018.5]
Flask-Cors==3.0.6
  - Flask [required: &gt;=0.9, installed: 1.0.2]
    - click [required: &gt;=5.1, installed: 6.7]
    - itsdangerous [required: &gt;=0.24, installed: 0.24]
    - Jinja2 [required: &gt;=2.10, installed: 2.10]
      - MarkupSafe [required: &gt;=0.23, installed: 1.0]
    - Werkzeug [required: &gt;=0.14, installed: 0.14.1]
  - Six [required: Any, installed: 1.11.0]
Flask-PyMongo==2.1.0
  - Flask [required: &gt;=0.11, installed: 1.0.2]
    - click [required: &gt;=5.1, installed: 6.7]
    - itsdangerous [required: &gt;=0.24, installed: 0.24]
    - Jinja2 [required: &gt;=2.10, installed: 2.10]
      - MarkupSafe [required: &gt;=0.23, installed: 1.0]
    - Werkzeug [required: &gt;=0.14, installed: 0.14.1]
  - PyMongo [required: &gt;=3.0, installed: 3.7.1]
Flask-RESTful==0.3.6
  - aniso8601 [required: &gt;=0.82, installed: 3.0.2]
  - Flask [required: &gt;=0.8, installed: 1.0.2]
    - click [required: &gt;=5.1, installed: 6.7]
    - itsdangerous [required: &gt;=0.24, installed: 0.24]
    - Jinja2 [required: &gt;=2.10, installed: 2.10]
      - MarkupSafe [required: &gt;=0.23, installed: 1.0]
    - Werkzeug [required: &gt;=0.14, installed: 0.14.1]
  - pytz [required: Any, installed: 2018.5]
  - six [required: &gt;=1.3.0, installed: 1.11.0]
gensim==3.5.0
  - numpy [required: &gt;=1.11.3, installed: 1.15.1]
  - scipy [required: &gt;=0.18.1, installed: 1.1.0]
    - numpy [required: &gt;=1.8.2, installed: 1.15.1]
  - six [required: &gt;=1.5.0, installed: 1.11.0]
  - smart-open [required: &gt;=1.2.1, installed: 1.6.0]
    - boto [required: &gt;=2.32, installed: 2.49.0]
    - boto3 [required: Any, installed: 1.8.1]
      - botocore [required: &gt;=1.11.1,&lt;1.12.0, installed: 1.11.1]
        - docutils [required: &gt;=0.10, installed: 0.14]
        - jmespath [required: &gt;=0.7.1,&lt;1.0.0, installed: 0.9.3]
        - python-dateutil [required: &gt;=2.1,&lt;3.0.0, installed: 2.7.3]
          - six [required: &gt;=1.5, installed: 1.11.0]
        - urllib3 [required: &lt;1.24,&gt;=1.20, installed: 1.23]
      - jmespath [required: &gt;=0.7.1,&lt;1.0.0, installed: 0.9.3]
      - s3transfer [required: &lt;0.2.0,&gt;=0.1.10, installed: 0.1.13]
        - botocore [required: &lt;2.0.0,&gt;=1.3.0, installed: 1.11.1]
          - docutils [required: &gt;=0.10, installed: 0.14]
          - jmespath [required: &gt;=0.7.1,&lt;1.0.0, installed: 0.9.3]
          - python-dateutil [required: &gt;=2.1,&lt;3.0.0, installed: 2.7.3]
            - six [required: &gt;=1.5, installed: 1.11.0]
          - urllib3 [required: &lt;1.24,&gt;=1.20, installed: 1.23]
    - bz2file [required: Any, installed: 0.98]
    - requests [required: Any, installed: 2.19.1]
      - certifi [required: &gt;=2017.4.17, installed: 2018.8.24]
      - chardet [required: &gt;=3.0.2,&lt;3.1.0, installed: 3.0.4]
      - idna [required: &lt;2.8,&gt;=2.5, installed: 2.7]
      - urllib3 [required: &lt;1.24,&gt;=1.21.1, installed: 1.23]
gunicorn==19.9.0
jieba==0.39
openpyxl==2.5.5
  - et-xmlfile [required: Any, installed: 1.0.1]
  - jdcal [required: Any, installed: 1.4]
patchwork==1.0.1
  - fabric [required: &lt;3.0,&gt;=2.0, installed: 2.3.1]
    - cryptography [required: &gt;=1.1, installed: 2.3.1]
      - asn1crypto [required: &gt;=0.21.0, installed: 0.24.0]
      - cffi [required: !=1.11.3,&gt;=1.7, installed: 1.11.5]
        - pycparser [required: Any, installed: 2.18]
      - idna [required: &gt;=2.1, installed: 2.7]
      - six [required: &gt;=1.4.1, installed: 1.11.0]
    - invoke [required: &lt;2.0,&gt;=1.1, installed: 1.1.1]
    - paramiko [required: &gt;=2.4, installed: 2.4.1]
      - bcrypt [required: &gt;=3.1.3, installed: 3.1.4]
        - cffi [required: &gt;=1.1, installed: 1.11.5]
          - pycparser [required: Any, installed: 2.18]
        - six [required: &gt;=1.4.1, installed: 1.11.0]
      - cryptography [required: &gt;=1.5, installed: 2.3.1]
        - asn1crypto [required: &gt;=0.21.0, installed: 0.24.0]
        - cffi [required: !=1.11.3,&gt;=1.7, installed: 1.11.5]
          - pycparser [required: Any, installed: 2.18]
        - idna [required: &gt;=2.1, installed: 2.7]
        - six [required: &gt;=1.4.1, installed: 1.11.0]
      - pyasn1 [required: &gt;=0.1.7, installed: 0.4.4]
      - pynacl [required: &gt;=1.0.1, installed: 1.2.1]
        - cffi [required: &gt;=1.4.1, installed: 1.11.5]
          - pycparser [required: Any, installed: 2.18]
        - six [required: Any, installed: 1.11.0]
pycnnum==1.0.1
pylru==1.1.0
PyMySQL==0.9.2
  - cryptography [required: Any, installed: 2.3.1]
    - asn1crypto [required: &gt;=0.21.0, installed: 0.24.0]
    - cffi [required: !=1.11.3,&gt;=1.7, installed: 1.11.5]
      - pycparser [required: Any, installed: 2.18]
    - idna [required: &gt;=2.1, installed: 2.7]
    - six [required: &gt;=1.4.1, installed: 1.11.0]
python-dotenv==0.9.1
redis==2.10.6
schedule==0.5.0
SolrClient==0.2.1
  - kazoo [required: ==2.2.1, installed: 2.2.1]
    - six [required: Any, installed: 1.11.0]
  - requests [required: &gt;=2.2.1, installed: 2.19.1]
    - certifi [required: &gt;=2017.4.17, installed: 2018.8.24]
    - chardet [required: &gt;=3.0.2,&lt;3.1.0, installed: 3.0.4]
    - idna [required: &lt;2.8,&gt;=2.5, installed: 2.7]
    - urllib3 [required: &lt;1.24,&gt;=1.21.1, installed: 1.23]
tqdm==4.25.0
</code>

看看虚拟环境信息:

<code>(robotDemo-dwdcgdaG) [root@xx-general-01 robotDemo]# pipenv --where
/root/xx/web/server/robotDemo
(robotDemo-dwdcgdaG) [root@xx-general-01 robotDemo]# pipenv --py
/root/.local/share/virtualenvs/robotDemo-dwdcgdaG/bin/python
(robotDemo-dwdcgdaG) [root@xx-general-01 robotDemo]# pipenv --envs
The following environment variables can be set, to do various things:

  - PIPENV_DEFAULT_PYTHON_VERSION
  - PIPENV_PYTHON
  - PIPENV_NOSPIN
  - PIPENV_SHELL
  - PIPENV_USE_SYSTEM
  - PIPENV_MAX_DEPTH
  - PIPENV_PIPFILE
  - PIPENV_SKIP_VALIDATION
  - PIPENV_VIRTUALENV
  - PIPENV_MAX_SUBPROCESS
  - PIPENV_NO_INHERIT
  - PIPENV_DONT_USE_PYENV
  - PIPENV_COLORBLIND
  - PIPENV_DONT_LOAD_ENV
  - PIPENV_TIMEOUT
  - PIPENV_HIDE_EMOJIS
  - PIPENV_TEST_INDEX
  - PIPENV_VENV_IN_PROJECT
  - PIPENV_USE_HASHES
  - PIPENV_MAX_ROUNDS
  - PIPENV_SHELL_FANCY
  - PIPENV_INSTALL_TIMEOUT
  - PIPENV_CACHE_DIR
  - PIPENV_DOTENV_LOCATION
  - PIPENV_YES

You can learn more at:
http://docs.pipenv.org/advanced/#configuration-with-environment-variables
(robotDemo-dwdcgdaG) [root@xx-general-01 robotDemo]# pipenv --venv
/root/.local/share/virtualenvs/robotDemo-dwdcgdaG
(robotDemo-dwdcgdaG) [root@xx-general-01 robotDemo]# pipenv --version
pipenv, version 11.10.0
</code>

其中的Python解析器位置是:

<code>/root/.local/share/virtualenvs/robotDemo-dwdcgdaG/bin/python
</code>

对应的:

<code>(robotDemo-dwdcgdaG) [root@xx-general-01 robotDemo]# ll -lha /root/.local/share/virtualenvs/robotDemo-dwdcgdaG/bin/
total 3.9M
drwxr-xr-x 3 root root 4.0K Aug 27 18:13 .
drwxr-xr-x 6 root root 4.0K Aug 27 18:12 ..
-rw-r--r-- 1 root root 2.1K Aug 27 18:12 activate
-rw-r--r-- 1 root root 1.1K Aug 27 18:12 activate.csh
-rw-r--r-- 1 root root 2.2K Aug 27 18:12 activate.fish
-rw-r--r-- 1 root root 1.2K Aug 27 18:12 activate_this.py
-rwxr-xr-x 1 root root  12K Aug 27 18:13 asadmin
-rwxr-xr-x 1 root root 1.6K Aug 27 18:13 bundle_image
-rwxr-xr-x 1 root root  266 Aug 27 18:12 celery
-rwxr-xr-x 1 root root 3.5K Aug 27 18:13 cfadmin
-rwxr-xr-x 1 root root  273 Aug 27 18:13 chardetect
-rwxr-xr-x 1 root root 3.1K Aug 27 18:13 cq
-rwxr-xr-x 1 root root 5.0K Aug 27 18:13 cwutil
-rwxr-xr-x 1 root root  259 Aug 27 18:13 dotenv
-rwxr-xr-x 1 root root 2.2K Aug 27 18:13 dynamodb_dump
-rwxr-xr-x 1 root root 3.5K Aug 27 18:13 dynamodb_load
-rwxr-xr-x 1 root root  282 Aug 27 18:12 easy_install
-rwxr-xr-x 1 root root  282 Aug 27 18:12 easy_install-3.6
-rwxr-xr-x 1 root root 9.6K Aug 27 18:13 elbadmin
-rwxr-xr-x 1 root root  806 Aug 27 18:13 f2py
-rwxr-xr-x 1 root root  272 Aug 27 18:13 fab
-rwxr-xr-x 1 root root 1.9K Aug 27 18:13 fetch_file
-rwxr-xr-x 1 root root  260 Aug 27 18:13 flask
-rwxr-xr-x 1 root root 5.2K Aug 27 18:13 glacier
-rwxr-xr-x 1 root root  269 Aug 27 18:13 gunicorn
-rwxr-xr-x 1 root root  271 Aug 27 18:13 gunicorn_paster
-rwxr-xr-x 1 root root 5.7K Aug 27 18:13 instance_events
-rwxr-xr-x 1 root root  272 Aug 27 18:12 inv
-rwxr-xr-x 1 root root  272 Aug 27 18:12 invoke
-rwxr-xr-x 1 root root 1.7K Aug 27 18:12 jp.py
-rwxr-xr-x 1 root root  970 Aug 27 18:13 kill_instance
-rwxr-xr-x 1 root root  11K Aug 27 18:13 launch_instance
-rwxr-xr-x 1 root root 3.1K Aug 27 18:13 list_instances
-rwxr-xr-x 1 root root 3.5K Aug 27 18:13 lss3
-rwxr-xr-x 1 root root  19K Aug 27 18:13 mturk
-rwxr-xr-x 1 root root  264 Aug 27 18:12 pip
-rwxr-xr-x 1 root root  264 Aug 27 18:12 pip3
-rwxr-xr-x 1 root root  264 Aug 27 18:12 pip3.6
-rwxr-xr-x 1 root root 2.7K Aug 27 18:13 pyami_sendmail
drwxr-xr-x 2 root root 4.0K Aug 27 18:13 __pycache__
lrwxrwxrwx 1 root root   10 Aug 27 18:12 python -&gt; python3.6m
lrwxrwxrwx 1 root root   10 Aug 27 18:12 python3 -&gt; python3.6m
lrwxrwxrwx 1 root root   10 Aug 27 18:12 python3.6 -&gt; python3.6m
-rwxr-xr-x 1 root root 3.6M Aug 27 18:12 python3.6m
-rwxr-xr-x 1 root root 2.4K Aug 27 18:12 python-config
-rwxr-xr-x 1 root root 8.9K Aug 27 18:13 route53
-rwxr-xr-x 1 root root  762 Aug 27 18:13 rst2html4.py
-rwxr-xr-x 1 root root 1.2K Aug 27 18:13 rst2html5.py
-rwxr-xr-x 1 root root  642 Aug 27 18:13 rst2html.py
-rwxr-xr-x 1 root root  839 Aug 27 18:13 rst2latex.py
-rwxr-xr-x 1 root root  648 Aug 27 18:13 rst2man.py
-rwxr-xr-x 1 root root 1.8K Aug 27 18:13 rst2odt_prepstyles.py
-rwxr-xr-x 1 root root  812 Aug 27 18:13 rst2odt.py
-rwxr-xr-x 1 root root  649 Aug 27 18:13 rst2pseudoxml.py
-rwxr-xr-x 1 root root  685 Aug 27 18:13 rst2s5.py
-rwxr-xr-x 1 root root  919 Aug 27 18:13 rst2xetex.py
-rwxr-xr-x 1 root root  650 Aug 27 18:13 rst2xml.py
-rwxr-xr-x 1 root root  718 Aug 27 18:13 rstpep2html.py
-rwxr-xr-x 1 root root  17K Aug 27 18:13 s3put
-rwxr-xr-x 1 root root 6.9K Aug 27 18:13 sdbadmin
-rwxr-xr-x 1 root root 3.8K Aug 27 18:13 taskadmin
-rwxr-xr-x 1 root root  261 Aug 27 18:12 tqdm
-rwxr-xr-x 1 root root  261 Aug 27 18:12 wheel
</code>

也有要的:celery和gunicorn

也先去用Fabric:

deploy/fabfile.py

<code>from invoke import task
from fabric import Connection
from patchwork.transfers import rsync
import os

RemoteHost = '47.x.x.x'
RemoteUser = 'xxx'
# RemotePathRoot = '/root/xx/nlp/test_fabric'
RemotePathRoot = '/root/xx'

def seperatorLine(seperatorChar='-', count=80):
    print(seperatorChar * count)

def fabLoadedPath():
    from fabric.main import program
    return program.collection.loaded_from

@task
def upload(context):
    print("upload: context=%s", context)
    seperatorLine()
    fabFilePath = fabLoadedPath()
    print("fabFilePath=%s" % fabFilePath)
    localProjectRootPath = os.path.join(fabFilePath, "..")
    print("localProjectRootPath=%s" % localProjectRootPath)
    seperatorLine()
    print("Local environment:")
    context.run("uname -a")
    context.run("pwd")
    context.run("ls -lha")
    seperatorLine()
    remoteConn = Connection(host=RemoteHost, user=RemoteUser)
    print(remoteConn)
    seperatorLine()
    print("Remote Server:")
    remoteConn.run('uname -a')
    remoteConn.run('pwd')
    print("remote path: %s" % RemotePathRoot)
    remoteConn.run('ls -lha %s' % (RemotePathRoot))
    # remoteConn.run('cd %s &amp;&amp; pwd &amp;&amp; ls -lha' % RemotePathRoot)
    # putFileResult = remoteConn.put('fabfile.py', remote=RemotePathRoot)
    # print("Uploaded {0.local} to {0.remote}".format(putFileResult))

    # syncSource = "../deploy"
    syncSource = localProjectRootPath
    syncTarget = RemotePathRoot
    # syncExclude = [".DS_Store", "data/", "processData/", "*.log", "*.pyc", "__pycache__"]
    syncExclude = [".DS_Store", "data/", "processData/mysqlQa/output_summary/", "*.log", "*.pyc", "__pycache__"]
    syncResp = rsync(
        remoteConn,
        source=syncSource,
        target=syncTarget,
        # Note: be careful to add `delete`, to void to delete unexpected files
        # delete=True,
        exclude=syncExclude)
    print("Sync lcoal %s to remote %s while exclude %s -&gt; return %s" %
          (syncSource, syncTarget, syncExclude, syncResp))
</code>

上传x的代码(其中包含用到的search):

<code>cd /Users/crifan/dev/dev_root/xxx
fab -r deploy upload
</code>

再去拷贝glove.6B数据到data中:

<code>(robotDemo-dwdcgdaG) [root@xx-general-01 xx]# cd /opt/word2vec/glove.6B
(robotDemo-dwdcgdaG) [root@x-general-01 glove.6B]# ll
total 3378124
-rwxrwxrwx 1 root root  347116733 Jul 18 21:52 glove.6B.100d.txt
-rwxrwxrwx 1 root root  693432828 Jul 18 21:57 glove.6B.200d.txt
-rwxrwxrwx 1 root root 1037962819 Jul 18 21:43 glove.6B.300d.txt
-rwxrwxrwx 1 root root 1037962830 Jul 18 21:50 glove.6B.300d.w2vformat.txt
-rwxrwxrwx 1 root root  171350079 Jul 18 21:53 glove.6B.50d.txt
-rwxrwxrwx 1 root root  171350089 Jul 18 21:44 glove.6B.50d.w2vformat.txt
(robotDemo-dwdcgdaG) [root@x-general-01 glove.6B]# ls -lha
total 3.3G
drwxrwxrwx 2 root root 4.0K Jul 18 21:53 .
drwxrwxrwx 3 root root 4.0K Jul 18 21:37 ..
-rwxrwxrwx 1 root root 332M Jul 18 21:52 glove.6B.100d.txt
-rwxrwxrwx 1 root root 662M Jul 18 21:57 glove.6B.200d.txt
-rwxrwxrwx 1 root root 990M Jul 18 21:43 glove.6B.300d.txt
-rwxrwxrwx 1 root root 990M Jul 18 21:50 glove.6B.300d.w2vformat.txt
-rwxrwxrwx 1 root root 164M Jul 18 21:53 glove.6B.50d.txt
-rwxrwxrwx 1 root root 164M Jul 18 21:44 glove.6B.50d.w2vformat.txt
(robotDemo-dwdcgdaG) [root@x-general-01 glove.6B]# cp /opt/word2vec/glove.6B/* /root/x/data/^C
(robotDemo-dwdcgdaG) [root@x-general-01 glove.6B]# pwd
/opt/word2vec/glove.6B
(robotDemo-dwdcgdaG) [root@x-general-01 glove.6B]# cd /root/x/data
(robotDemo-dwdcgdaG) [root@x-general-01 data]# ll
total 0
(robotDemo-dwdcgdaG) [root@x-general-01 data]# mkdir glove.6B
(robotDemo-dwdcgdaG) [root@x-general-01 data]# cp /opt/word2vec/glove.6B/* /root/x/data/glove.6B/
(robotDemo-dwdcgdaG) [root@x-general-01 data]# pwd
/root/x/data
(robotDemo-dwdcgdaG) [root@x-general-01 data]# cd glove.6B/
(robotDemo-dwdcgdaG) [root@x-general-01 glove.6B]# ll -lha
total 3.3G
drwxr-xr-x 2 root root 4.0K Aug 27 17:43 .
drwxr-xr-x 3 root root 4.0K Aug 27 17:43 ..
-rwxr-xr-x 1 root root 332M Aug 27 17:43 glove.6B.100d.txt
-rwxr-xr-x 1 root root 662M Aug 27 17:43 glove.6B.200d.txt
-rwxr-xr-x 1 root root 990M Aug 27 17:43 glove.6B.300d.txt
-rwxr-xr-x 1 root root 990M Aug 27 17:43 glove.6B.300d.w2vformat.txt
-rwxr-xr-x 1 root root 164M Aug 27 17:43 glove.6B.50d.txt
-rwxr-xr-x 1 root root 164M Aug 27 17:43 glove.6B.50d.w2vformat.txt
</code>

然后再去更新相关的

gunicorn和supervisor的配置:

更新了配置文件后:

conf/gunicorn/gunicorn_config.py

<code>import multiprocessing
import os
import sys
# sys.path.append(".")
# from config import BaseConfig

curPath = os.path.dirname(os.path.abspath(__file__))
print("curPath=%s" % curPath)
confPath = os.path.join(curPath, "..")
print("confPath=%s" % confPath)
sys.path.append(confPath)
from conf.app import settings
print("settings=%s" % settings)

appRootPath = os.path.join(confPath, "..")
print("appRootPath=%s" % appRootPath)
#appRootPath = "/Users/crifan/dev/dev_root/xxx/robotDemo"
# appRootPath = os.getcwd()
# print("appRootPath=%s" % appRootPath)
# flaskHost = BaseConfig.FLASK_HOST
# flaskPort = BaseConfig.FLASK_PORT
flaskHost = settings.FLASK_HOST
flaskPort = settings.FLASK_PORT
print("flaskHost=%s, flaskPort=%s" % (flaskHost, flaskPort))


reload = True                                   #当代码改变时自动重启服务
#bind = '127.0.0.1:32851'                        #绑定ip和端口号
# bind = '0.0.0.0:32851'                        #绑定ip和端口号
bind = ("%s:%s" % (flaskHost, flaskPort))
backlog = 512                                   #监听队列
chdir = appRootPath                             #gunicorn要切换到的目的工作目录
timeout = 30                                    #超时
worker_class = 'sync'                           #默认的是sync模式
workers = multiprocessing.cpu_count() * 2 + 1   #进程数
threads = 2                                     #指定每个进程开启的线程数
loglevel = 'info'                               #日志级别,这个日志级别指的是错误日志的级别,而访问日志的级别无法设置
access_log_format = '%(t)s %(p)s %(h)s "%(r)s" %(s)s %(L)s %(b)s %(f)s" "%(a)s"'    #设置gunicorn访问日志格式,错误日志无法设置
"""
其每个选项的含义如下:
h          remote address
l          '-'
u          currently '-', may be user name in future releases
t          date of the request
r          status line (e.g. ``GET / HTTP/1.1``)
s          status
b          response length or '-'
f          referer
a          user agent
T          request time in seconds
D          request time in microseconds
L          request time in decimal seconds
p          process ID
"""

accesslog = appRootPath + "/logs/gunicorn_access.log"      #访问日志文件
errorlog  = appRootPath + "/logs/gunicorn_error.log"       #错误日志文件
</code>

和:

conf/supervisor/supervisord_server.conf

<code>;ROBOTDEMO_PRJOECT_ROOT=/root/x/web/server/robotDemo
;ROBOTDEMO_VIRTUALENV_ROOT=/root/.local/share/virtualenvs/robotDemo-dwdcgdaG

[program:redis]
directory=/root/x/web/server/robotDemo
;command=/usr/bin/redis-server
command=/usr/bin/redis-server /etc/redis.conf --dir /root/x/x/server/robotDemo/runtime

autostart=true
autorestart=true

stdout_logfile=/root/x/web/server/robotDemo/logs/redis-%(program_name)s-stdout.log
stdout_logfile_maxbytes=2MB
stdout_logfile_backups=10

stderr_logfile=/root/x/web/server/robotDemo/logs/redis-%(program_name)s-stderr.log
stderr_logfile_maxbytes=2MB
stderr_logfile_backups=10


[program:robotDemo_CeleryWorker]
command=/root/.local/share/virtualenvs/robotDemo-dwdcgdaG/bin/celery worker -A resources.tasks.celery
directory=/root/x/web/server/robotDemo
autostart=true
autorestart=true

stdout_logfile=/root/x/web/server/robotDemo/logs/celery-worker-%(program_name)s-stdout.log
stdout_logfile_maxbytes=2MB
stdout_logfile_backups=10

stderr_logfile=/root/x/web/server/robotDemo/logs/celery-worker-%(program_name)s-stderr.log
stderr_logfile_maxbytes=2MB
stderr_logfile_backups=10


[program:robotDemo_CeleryBeat]
command=/root/.local/share/virtualenvs/robotDemo-dwdcgdaG/bin/celery beat -A resources.tasks.celery -s /root/x/web/server/robotDemo/runtime/celerybeat-schedule
directory=/root/x/web/server/robotDemo
autostart=true
autorestart=true

stdout_logfile=/root/x/web/server/robotDemo/logs/celery-beat-%(program_name)s-stdout.log
stdout_logfile_maxbytes=2MB
stdout_logfile_backups=10

stderr_logfile=/root/x/web/server/robotDemo/logs/celery-beat-%(program_name)s-stderr.log
stderr_logfile_maxbytes=2MB
stderr_logfile_backups=10


[program:robotDemo]
command=/root/.local/share/virtualenvs/robotDemo-dwdcgdaG/bin/gunicorn -c conf/gunicorn/gunicorn_config.py app:app
environment = FLASK_ENV="production"
directory=/root/x/web/server/robotDemo
startsecs=0
stopwaitsecs=0
autostart=true
autorestart=true
killasgroup=true
stopasgroup=true

stdout_logfile=/root/x/web/server/robotDemo/logs/supervisord-%(program_name)s-stdout.log
stdout_logfile_maxbytes=2MB
stdout_logfile_backups=10

stderr_logfile=/root/x/web/server/robotDemo/logs/supervisord-%(program_name)s-stderr.log
stderr_logfile_maxbytes=2MB
stderr_logfile_backups=10
</code>

用fabric部署更新文件到服务器,然后再去服务器中:

<code>(robotDemo-dwdcgdaG) [root@x-general-01 robotDemo]# cp -f conf/supervisor/supervisord_server.conf /etc/supervisord.d/supervisord_server.conf
cp: overwrite ‘/etc/supervisord.d/supervisord_server.conf’? y
(robotDemo-dwdcgdaG) [root@x-general-01 robotDemo]# supervisorctl reload
Restarted supervisord
(robotDemo-dwdcgdaG) [root@x-general-01 robotDemo]# supervisorctl status
gunicorn                         STOPPED   Not started
redis                            RUNNING   pid 15818, uptime 0:00:02
robotDemo                        RUNNING   pid 15816, uptime 0:00:02
robotDemo_CeleryBeat             RUNNING   pid 15817, uptime 0:00:02
robotDemo_CeleryWorker           RUNNING   pid 15819, uptime 0:00:02
(robotDemo-dwdcgdaG) [root@x-general-01 robotDemo]# supervisorctl start redis gunicorn                         STOPPED   Not started^C
(robotDemo-dwdcgdaG) [root@x-general-01 robotDemo]# supervisorctl start redis robotDemo robotDemo_CeleryBeat robotDemo_CeleryWorker
redis: ERROR (already started)
robotDemo: ERROR (already started)
robotDemo_CeleryBeat: ERROR (already started)
robotDemo_CeleryWorker: ERROR (already started)
</code>

结果在线测试失败:

睿小麟英语智能机器人演示

http://xx/robotDemo/

需要找找原因。

<code>[root@xx-general-01 robotDemo]# supervisorctl status
gunicorn                         STOPPED   Not started
redis                            RUNNING   pid 15818, uptime 14:38:53
robotDemo                        RUNNING   pid 9723, uptime 0:00:00
robotDemo_CeleryBeat             RUNNING   pid 15817, uptime 14:38:53
robotDemo_CeleryWorker           RUNNING   pid 15819, uptime 14:38:53
[root@x-general-01 robotDemo]# supervisorctl reload
Restarted supervisord
[root@x-general-01 robotDemo]# supervisorctl status
error: &lt;class 'xmlrpclib.Fault'&gt;, &lt;Fault 6: 'SHUTDOWN_STATE'&gt;: file: /usr/lib64/python2.7/xmlrpclib.py line: 794
[root@xx-general-01 robotDemo]# supervisorctl status
gunicorn                         STOPPED   Not started
redis                            STARTING  
robotDemo                        RUNNING   pid 9731, uptime 0:00:00
robotDemo_CeleryBeat             STARTING  
robotDemo_CeleryWorker           STARTING  
[root@xx-general-01 robotDemo]# supervisorctl status
gunicorn                         STOPPED   Not started
redis                            RUNNING   pid 9733, uptime 0:00:02
robotDemo                        RUNNING   pid 9731, uptime 0:00:02
robotDemo_CeleryBeat             RUNNING   pid 9732, uptime 0:00:02
robotDemo_CeleryWorker           RUNNING   pid 9734, uptime 0:00:02
</code>

看到状态有些奇怪:

gunicorn是另外一个项目,状态是stopped

而之前没有改动这个项目,应该是正常运行才对

去重启:

<code>[root@x-general-01 robotDemo]# supervisorctl restart all
robotDemo: stopped
robotDemo_CeleryBeat: stopped
redis: stopped
robotDemo_CeleryWorker: stopped
robotDemo: started
robotDemo_CeleryBeat: started
redis: started
gunicorn: started
robotDemo_CeleryWorker: started
[root@x-general-01 robotDemo]# supervisorctl status     
gunicorn                         RUNNING   pid 9796, uptime 0:00:05
redis                            RUNNING   pid 9795, uptime 0:00:05
robotDemo                        RUNNING   pid 9793, uptime 0:00:05
robotDemo_CeleryBeat             RUNNING   pid 9794, uptime 0:00:05
robotDemo_CeleryWorker           RUNNING   pid 9797, uptime 0:00:05
</code>

然后删除log再去重启:

<code>[root@x-general-01 robotDemo]# pwd
/root/x/web/server/robotDemo
[root@xx-general-01 robotDemo]# ll
total 92
-rw-r--r-- 1 root root  1448 Aug 27 10:26 app.py
drwxr-xr-x 3 root root  4096 Aug 27 18:25 common
drwxr-xr-x 6 root root  4096 Aug 27 18:25 conf
drwxr-xr-x 2 root root  4096 Aug 27 18:03 deploy
-rw-r--r-- 1 root root  6707 Aug 25 15:07 factory.py
drwxr-xr-x 2 root root  4096 Aug 28 04:18 logs
-rw-r--r-- 1 root root   495 Aug 27 10:35 Pipfile
-rw-r--r-- 1 root root 35942 Aug 27 18:12 Pipfile.lock
drwxr-xr-x 2 root root  4096 Aug 27 18:25 __pycache__
-rw-r--r-- 1 root root  6267 Aug 27 14:03 README.md
drwxr-xr-x 3 root root  4096 Aug 27 18:25 resources
drwxr-xr-x 2 root root  4096 Aug 28 09:06 runtime
drwxr-xr-x 3 root root  4096 Aug 27 18:25 tmp
[root@x-general-01 robotDemo]# rm -rf logs
[root@x-general-01 robotDemo]# mkdir logs
[root@x-general-01 robotDemo]# supervisorctl restart all
robotDemo: started
robotDemo_CeleryBeat: started
redis: started
gunicorn: started
robotDemo_CeleryWorker: started
[root@x-general-01 robotDemo]# supervisorctl status
gunicorn                         RUNNING   pid 9843, uptime 0:00:07
redis                            RUNNING   pid 9842, uptime 0:00:07
robotDemo                        RUNNING   pid 9870, uptime 0:00:01
robotDemo_CeleryBeat             RUNNING   pid 9841, uptime 0:00:07
robotDemo_CeleryWorker           RUNNING   pid 9844, uptime 0:00:07
</code>

方便去查看启动的log是否正常。

【未解决】在线环境中用gunicorn部署的产品demo无法正常初始化运行

转载请注明:在路上 » 【记录】把合并了基于搜索的兜底对话的产品demo部署到在线环境中

发表我的评论
取消评论

表情

Hi,您需要填写昵称和邮箱!

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址
82 queries in 0.203 seconds, using 22.25MB memory