最新消息:20190717 VPS服务器:Vultr新加坡,WordPress主题:大前端D8,统一介绍入口:关于

【已解决】mac中PySpider运行出错:Deprecated option domaincontroller use http_authenticator.domain_controller instead

HTTP crifan 67浏览 0评论
折腾:
【已解决】PySpider中用Python代码爬取小花生app中绘本数据
期间,在
pipenv install
去安装PySpider之后,进入虚拟环境运行,结果报错:
pyspider -c config.json 
phantomjs fetcher running on port 25555
[I 190327 16:32:30 result_worker:49] result_worker starting...
[I 190327 16:32:30 processor:211] processor starting...
[I 190327 16:32:31 tornado_fetcher:638] fetcher starting...
[I 190327 16:32:31 scheduler:647] scheduler starting...
[I 190327 16:32:31 scheduler:782] scheduler.xmlrpc listening on 127.0.0.1:23333
[I 190327 16:32:31 scheduler:586] in 5m: new:0,success:0,retry:0,failed:0
[I 190327 16:32:31 app:84] webui exiting...
Traceback (most recent call last):
  File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/bin/pyspider", line 10, in <module>
    sys.exit(main())
  File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/pyspider/run.py", line 754, in main
    cli()
  File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/click/core.py", line 1114, in invoke
    return Command.invoke(self, ctx)
  File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/pyspider/run.py", line 165, in cli
    ctx.invoke(all)
  File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/pyspider/run.py", line 497, in all
    ctx.invoke(webui, **webui_config)
  File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/pyspider/run.py", line 384, in webui
    app.run(host=host, port=port)
  File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/pyspider/webui/app.py", line 59, in run
    from .webdav import dav_app
  File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/pyspider/webui/webdav.py", line 216, in <module>
    dav_app = WsgiDAVApp(config)
  File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/wsgidav/wsgidav_app.py", line 135, in __init__
    _check_config(config)
  File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/wsgidav/wsgidav_app.py", line 119, in _check_config
    raise ValueError("Invalid configuration:\n  - " + "\n  - ".join(errors))
ValueError: Invalid configuration:
  - Deprecated option 'domaincontroller': use 'http_authenticator.domain_controller' instead.
http_authenticator.domain_controller
python – pyspider 启动错误 – SegmentFault 思否
结果没有wsgidav:
pipenv uninstall wsgidav
Uninstalling wsgidav...
No package wsgidav to remove from Pipfile.
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
pyspider Deprecated option domaincontroller use http_authenticator.domain_controller instead
PySpider:一个国人编写的强大的网络爬虫系统并带有强大的WebUI – Python开发 – 评论 | CTOLib码库
Error to start webui service · Issue #831 · binux/pyspider
“WsgiDAV released a new version — pre-release 3.x
‘dir_browser.enable’ DEPRECATED with 2.4.0
and ‘domaincontroller’ updated to ‘http_authenticator.HTTPAuthenticator’
reference:https://wsgidav.readthedocs.io/en/latest/addons-ntdc.html
You can change pyspider/webui/webdav.py line 203:
config = DEFAULT_CONFIG.copy()
 config.update({
    'mount_path': '/dav',
    'provider_mapping': {
        '/': ScriptProvider(app)
    },
    #'domaincontroller': NeedAuthController(app),
    'http_authenticator': {
        'HTTPAuthenticator':NeedAuthController(app),
    },
    
    'verbose': 1 if app.debug else 0,
    
    'dir_browser': {'davmount': False,
                    #'enable': True,
                    'msmount': False,
                    'response_trailer': ''},
})
dav_app = WsgiDAVApp(config)
去修改一下
pyspider安装遇到的坑记录 – sinkpuer的博客 – CSDN博客
不过打算直接先去安装2.4.1的wsgidav
pipenv reinstall wsgidav==2.4.1
Usage: pipenv [OPTIONS] COMMAND [ARGS]...
Try "pipenv --help" for help.

Error: No such command "reinstall".

Did you mean one of these?
    install
    uninstall
➜  crawler_xiaohuasheng_app git:(master) ✗ pipenv install wsgidav==2.4.1 
Installing wsgidav==2.4.1...
Adding wsgidav to Pipfile's [packages]...
✔ Installation Succeeded 
Pipfile.lock not found, creating...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
结果:
就可以了:
pyspider -c config.json        
Error: Could not create web server listening on port 25555
[I 190327 16:40:32 result_worker:49] result_worker starting...
[I 190327 16:40:33 processor:211] processor starting...
[I 190327 16:40:33 tornado_fetcher:638] fetcher starting...
[I 190327 16:40:33 scheduler:647] scheduler starting...
[I 190327 16:40:33 scheduler:782] scheduler.xmlrpc listening on 127.0.0.1:23333
[I 190327 16:40:33 scheduler:586] in 5m: new:0,success:0,retry:0,failed:0
[I 190327 16:40:33 app:76] webui running on 0.0.0.0:7700
Error: Could not create web server listening on port 25555
Error: Could not create web server listening on port 25555
Error: Could not create web server listening on port 25555
Error: Could not create web server listening on port 25555
^C[I 190327 16:40:43 app:84] webui exiting...
[I 190327 16:40:43 scheduler:663] scheduler exiting...
[I 190327 16:40:43 tornado_fetcher:671] fetcher exiting...
[I 190327 16:40:43 processor:229] processor exiting...
[I 190327 16:40:43 result_worker:66] result_worker exiting...

Aborted!
而此处估计是之前的端口被占用了,去看看:
lsof -i:25555
COMMAND     PID   USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
phantomjs 62688 crifan   11u  IPv4 0xc72d8eb51fef7cb9      0t0  TCP *:25555 (LISTEN)
➜  crawler_xiaohuasheng_app git:(master) ✗ kill -9 62688
➜  crawler_xiaohuasheng_app git:(master) ✗ lsof -i:25555
即可正常运行了:
pyspider -c config.json
phantomjs fetcher running on port 25555
[I 190327 16:42:07 result_worker:49] result_worker starting...
[I 190327 16:42:07 processor:211] processor starting...
[I 190327 16:42:08 tornado_fetcher:638] fetcher starting...
[I 190327 16:42:08 scheduler:647] scheduler starting...
[I 190327 16:42:08 scheduler:782] scheduler.xmlrpc listening on 127.0.0.1:23333
[I 190327 16:42:08 scheduler:586] in 5m: new:0,success:0,retry:0,failed:0
[I 190327 16:42:08 app:76] webui running on 0.0.0.0:7700
【总结】
此处运行PySpider会出错:
lib/python3.6/site-packages/wsgidav/wsgidav_app.py", line 119, in _check_config
    raise ValueError("Invalid configuration:\n  - " + "\n  - ".join(errors))
ValueError: Invalid configuration:
  - Deprecated option 'domaincontroller': use 'http_authenticator.domain_controller' instead.
原因是:
wsgidav库升级到了最新版,导出不兼容。
解决办法:
安装旧版本:
pipenv install wsgidav==2.4.1
或:
手动改代码为:
pyspider/webui/webdav.py 203行:
 config = DEFAULT_CONFIG.copy()
 config.update({
    'mount_path': '/dav',
    'provider_mapping': {
        '/': ScriptProvider(app)
    },
    #'domaincontroller': NeedAuthController(app),
    'http_authenticator': {
        'HTTPAuthenticator':NeedAuthController(app),
    },
    
    'verbose': 1 if app.debug else 0,
    
    'dir_browser': {'davmount': False,
                    #'enable': True,
                    'msmount': False,
                    'response_trailer': ''},
})
dav_app = WsgiDAVApp(config)

转载请注明:在路上 » 【已解决】mac中PySpider运行出错:Deprecated option domaincontroller use http_authenticator.domain_controller instead

发表我的评论
取消评论

表情

Hi,您需要填写昵称和邮箱!

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址
16 queries in 0.089 seconds, using 9.42MB memory