最新消息:20190717 VPS服务器:Vultr新加坡,WordPress主题:大前端D8,统一介绍入口:关于

【已解决】同步导出在线Mongo中某个集合(表)到本地

MongoDB crifan 696浏览 0评论

有个在线的Mongo数据库,其中有个dialog的字表,集合。

想要导出,导入本地的Mongo中,用于开发测试。

mongo 导出

mongodb 备份、还原、导入、导出简单操作 – 开发笔记 – SegmentFault 思否

好像用:

  • mongodump

  • mongorestore

即可。

mongodb数据导入导出以及备份恢复 – CSDN博客

MongoDB导入导出以及数据库备份 – CSDN博客

提到了:

  • mongoimport

MongoDB导入导出以及数据库备份 · Issue #22 · webplus/blog

  • mongoexport

  • mongoimport

和dump,restore有何区别?

【已解决】mongoimport/mongoexport和mongodump/mongorestore的区别

mongodb 备份 还原 导出 导入«海底苍鹰(tank)博客

Mongodb数据导出工具mongoexport和导入工具mongoimport介绍 – 青葱岁月 – ITeye博客

MongoDB数据导入导出 – 简书

好像相对新的帖子,都是推荐:

  • mongoexport

  • mongoimport

mongodb export and import

Import and Export Data — MongoDB Compass stable

说的是:

MongoDB Compass

的工具,支持导入和导出,支持JSON和CSV格式

-》抽空去试试:MONGODB COMPASS

mongoexport — MongoDB Manual

再去看看:

mongoimport — MongoDB Manual

而关于备份,详见官网:

MongoDB Backup Methods — MongoDB Manual

-》

https://docs.mongodb.com/manual/core/backups/#back-up-with-mongodump

数据导入和导出 — MongoDB Manual

https://mongodb-documentation.readthedocs.io/en/latest/administration/import-export.html

参考具体的使用介绍:

https://docs.mongodb.com/manual/reference/program/mongoimport/#use

https://docs.mongodb.com/manual/reference/program/mongoexport/#use

然后去试试

➜  backup_restore mongoexport –help

Usage:

  mongoexport <options>

Export data from MongoDB in CSV or JSON format.

See http://docs.mongodb.org/manual/reference/program/mongoexport/ for more information.

general options:

      –help                                      print usage

      –version                                   print the tool version and exit

verbosity options:

  -v, –verbose=<level>                           more detailed log output (include multiple times for more verbosity, e.g. -vvvvv, or specify a numeric value,

                                                  e.g. –verbose=N)

      –quiet                                     hide all log output

connection options:

  -h, –host=<hostname>                           mongodb host to connect to (setname/host1,host2 for replica sets)

      –port=<port>                               server port (can also use –host hostname:port)

ssl options:

      –ssl                                       connect to a mongod or mongos that has ssl enabled

      –sslCAFile=<filename>                      the .pem file containing the root certificate chain from the certificate authority

      –sslPEMKeyFile=<filename>                  the .pem file containing the certificate and key

      –sslPEMKeyPassword=<password>              the password to decrypt the sslPEMKeyFile, if necessary

      –sslCRLFile=<filename>                     the .pem file containing the certificate revocation list

      –sslAllowInvalidCertificates               bypass the validation for server certificates

      –sslAllowInvalidHostnames                  bypass the validation for server name

      –sslFIPSMode                               use FIPS mode of the installed openssl library

authentication options:

  -u, –username=<username>                       username for authentication

  -p, –password=<password>                       password for authentication

      –authenticationDatabase=<database-name>    database that holds the user’s credentials

      –authenticationMechanism=<mechanism>       authentication mechanism to use

namespace options:

  -d, –db=<database-name>                        database to use

  -c, –collection=<collection-name>              collection to use

uri options:

      –uri=mongodb-uri                           mongodb uri connection string

output options:

  -f, –fields=<field>[,<field>]*                 comma separated list of field names (required for exporting CSV) e.g. -f "name,age"

      –fieldFile=<filename>                      file with field names – 1 per line

      –type=<type>                               the output format, either json or csv (defaults to ‘json’) (default: json)

  -o, –out=<filename>                            output file; if not specified, stdout is used

      –jsonArray                                 output to a JSON array rather than one object per line

      –pretty                                    output JSON formatted to be human-readable

      –noHeaderLine                              export CSV data without a list of field names at the first line

querying options:

  -q, –query=<json>                              query filter, as a JSON string, e.g., ‘{x:{$gt:1}}’

      –queryFile=<filename>                      path to a file containing a query filter (JSON)

  -k, –slaveOk                                   allow secondary reads if available (default true) (default: false)

      –readPreference=<string>|<json>            specify either a preference name or a preference json object

      –forceTableScan                            force a table scan (do not use $snapshot)

      –skip=<count>                              number of documents to skip

      –limit=<count>                             limit the number of documents to export

      –sort=<json>                               sort order, as a JSON string, e.g. ‘{x:1}’

      –assertExists                              if specified, export fails if the collection does not exist (default: false)

然后去试试:

➜  backup_restore mongoexport -h ip_address –port PORT -u root -p password -d dialog -o mongodb_dialog.json

2018-06-29T11:38:29.852+0800    error connecting to db server: server returned error on SASL authentication step: Authentication failed.

➜  backup_restore mongoexport -h ip_address –port PORT -u root -p password –authenticationDatabase dialog -d dialog -o mongodb_dialog.json

2018-06-29T11:42:31.022+0800    error connecting to db server: server returned error on SASL authentication step: Authentication failed.

➜  backup_restore mongoexport -h ip_address –port PORT -u dialpg -p password –authenticationDatabase dialog -d dialog -o mongodb_dialog.json

2018-06-29T11:42:58.890+0800    error connecting to db server: server returned error on SASL authentication step: Authentication failed.

➜  backup_restore mongoexport -h ip_address –port PORT -u dialog -p password –authenticationDatabase dialog -d dialog -o mongodb_dialog.json

2018-06-29T11:43:06.318+0800    error validating settings: must specify a collection

2018-06-29T11:43:06.319+0800    try ‘mongoexport –help’ for more information

最后用:

mongoexport -h IP –port PORT -u dialog -p PASSWORD –authenticationDatabase dialog -d dialog -c dialog -o mongodb_dialog.json

可以了:

但是却一直是0%

➜  backup_restore mongoexport -h IP –port PORT -u dialog -p pwd –authenticationDatabase dialog -d dialog -c dialog -o mongodb_dialog.json

2018-06-29T11:44:45.057+0800    connected to: ip:port

2018-06-29T11:44:46.018+0800    [……………………]  dialog.dialog  0/41714  (0.0%)

2018-06-29T11:46:32.022+0800    [……………………]  dialog.dialog  0/41714  (0.0%)

2018-06-29T11:46:33.018+0800    [……………………]  dialog.dialog  0/41714  (0.0%)

2018-06-29T11:46:34.020+0800    [……………………]  dialog.dialog  0/41714  (0.0%)

^C2018-06-29T11:46:34.689+0800    signal ‘interrupt’ received; forcefully terminating

先去看看:

error validating settings: must specify a collection

mongoimport – MongoDB – validating settings error – Stack Overflow

Could not export whole database using MongoDB in Ubuntu – Stack Overflow

mongodb – mongoimport : ‘error validating settings: only one positional argument is allowed’ – Stack Overflow

mongoexport 0.0%

[TOOLS-1609] mongoexport fails on mac os 10.12.3 – MongoDB

➜  backup_restore mongo –version

MongoDB shell version v3.6.3

git version: 9586e557d54ef70f9ca4b43c26892cd55257e1a5

OpenSSL version: OpenSSL 1.0.2o  27 Mar 2018

allocator: system

modules: none

build environment:

    distarch: x86_64

    target_arch: x86_64

➜  backup_restore mongoexport –version

mongoexport version: 3.6.3

git version: homebrew

Go version: go1.10

   os: darwin

   arch: amd64

   compiler: gc

OpenSSL version: OpenSSL 1.0.2n  7 Dec 2017

此处版本是3.6.3,大于3.2.10,不存在兼容性问题。

[anspress] | MongoDB中文社区

好像继续多等等,就可以了?

mongodb Failed: error connecting to db server: no reachable servers – Stack Overflow

error connecting to db server: no reachable servers

MongoDB数据迁移 – TianHL – 博客园

不过想到了:

还是去在线服务器中导出,然后压缩导出的json文件,再下载文件,效率更高些。

结果发现在线服务器的本地中去export,速度也很慢:

-》或许是因为的确数据太多了,实际不慢但显得很慢?

的确是,稍等几十秒就可可以看到进度了:

[root@xx-general-01 mongo_backup_restore]# mongoexport –port pport -u dialog -p pwd –authenticationDatabase dialog -d dialog -c dialog -o mongodb_dialog_dialog.json

2018-06-29T13:31:20.841+0800    connected to: localhost:PORT

2018-06-29T13:31:21.841+0800    [……………………]  dialog.dialog  0/41714  (0.0%)

2018-06-29T13:31:51.841+0800    [####………………..]  dialog.dialog  8000/41714  (19.2%)…

2018-06-29T13:32:18.841+0800    [####………………..]  dialog.dialog  8000/41714  (19.2%)

2018-06-29T13:32:19.841+0800    [#########……………]  dialog.dialog  16000/41714  (38.4%)

2018-06-29T13:32:24.841+0800    [##################……]  dialog.dialog  32000/41714  (76.7%)

…32000/41714  (76.7%)

2018-06-29T13:32:49.841+0800    [##################……]  dialog.dialog  32000/41714  (76.7%)

2018-06-29T13:32:50.291+0800    [########################]  dialog.dialog  41714/41714  (100.0%)

2018-06-29T13:32:50.291+0800    exported 41714 records

果然是,导出后,500多MB:

[root@xx-general-01 mongo_backup_restore]# ll -ha

total 584M

drwxr-xr-x 2 root root 4.0K Jun 29 13:31 .

drwxr-xr-x 3 root root 4.0K Jun 29 13:31 ..

-rw-r–r– 1 root root 584M Jun 29 13:32 mongodb_dialog_dialog.json

去压缩,再下载:

-rw-r–r– 1 root root 584M Jun 29 13:32 mongodb_dialog_dialog.json

[root@xx-general-01 mongo_backup_restore]# 7za a -t7z -r -bt mongodb_dialog_dialog.json.7z mongodb_dialog_dialog.json

7-Zip (a) [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21

p7zip Version 16.02 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,64 bits,4 CPUs x64)

Scanning the drive:

1 file, 611756316 bytes (584 MiB)

Creating archive: mongodb_dialog_dialog.json.7z

Items to compress: 1

                                   

Files read from disk: 1

Archive size: 92200295 bytes (88 MiB)

Everything is Ok

[root@xx-general-01 mongo_backup_restore]# ll

total 687464

-rw-r–r– 1 root root 611756316 Jun 29 13:32 mongodb_dialog_dialog.json

-rw-r–r– 1 root root  92200295 Jun 29 13:39 mongodb_dialog_dialog.json.7z

[root@xx-general-01 mongo_backup_restore]# ll -lha

total 672M

drwxr-xr-x 2 root root 4.0K Jun 29 13:35 .

drwxr-xr-x 3 root root 4.0K Jun 29 13:31 ..

-rw-r–r– 1 root root 584M Jun 29 13:32 mongodb_dialog_dialog.json

-rw-r–r– 1 root root  88M Jun 29 13:39 mongodb_dialog_dialog.json.7z

[root@xx-general-01 mongo_backup_restore]# sz mongodb_dialog_dialog.json.7z

rz

Starting zmodem transfer.  Press Ctrl+C to cancel.

Transferring mongodb_dialog_dialog.json.7z…

  100%   90039 KB     119 KB/sec    00:12:36       0 Errors  

然后去导入本地mongo数据库:

➜  backup_restore mongoimport –help

Usage:

  mongoimport <options> <file>

Import CSV, TSV or JSON data into MongoDB. If no file is provided, mongoimport reads from stdin.

See http://docs.mongodb.org/manual/reference/program/mongoimport/ for more information.

general options:

      –help                                      print usage

      –version                                   print the tool version and exit

verbosity options:

  -v, –verbose=<level>                           more detailed log output (include multiple times for more verbosity, e.g. -vvvvv, or specify a numeric value,

                                                  e.g. –verbose=N)

      –quiet                                     hide all log output

connection options:

  -h, –host=<hostname>                           mongodb host to connect to (setname/host1,host2 for replica sets)

      –port=<port>                               server port (can also use –host hostname:port)

ssl options:

      –ssl                                       connect to a mongod or mongos that has ssl enabled

      –sslCAFile=<filename>                      the .pem file containing the root certificate chain from the certificate authority

      –sslPEMKeyFile=<filename>                  the .pem file containing the certificate and key

      –sslPEMKeyPassword=<password>              the password to decrypt the sslPEMKeyFile, if necessary

      –sslCRLFile=<filename>                     the .pem file containing the certificate revocation list

      –sslAllowInvalidCertificates               bypass the validation for server certificates

      –sslAllowInvalidHostnames                  bypass the validation for server name

      –sslFIPSMode                               use FIPS mode of the installed openssl library

authentication options:

  -u, –username=<username>                       username for authentication

  -p, –password=<password>                       password for authentication

      –authenticationDatabase=<database-name>    database that holds the user’s credentials

      –authenticationMechanism=<mechanism>       authentication mechanism to use

namespace options:

  -d, –db=<database-name>                        database to use

  -c, –collection=<collection-name>              collection to use

uri options:

      –uri=mongodb-uri                           mongodb uri connection string

input options:

  -f, –fields=<field>[,<field>]*                 comma separated list of fields, e.g. -f name,age

      –fieldFile=<filename>                      file with field names – 1 per line

      –file=<filename>                           file to import from; if not specified, stdin is used

      –headerline                                use first line in input source as the field list (CSV and TSV only)

      –jsonArray                                 treat input source as a JSON array

      –parseGrace=<grace>                        controls behavior when type coercion fails – one of: autoCast, skipField, skipRow, stop (defaults to ‘stop’)

                                                  (default: stop)

      –type=<type>                               input format to import: json, csv, or tsv (defaults to ‘json’) (default: json)

      –columnsHaveTypes                          indicated that the field list (from –fields, –fieldsFile, or –headerline) specifies types; They must be in

                                                  the form of ‘<colName>.<type>(<arg>)’. The type can be one of: auto, binary, bool, date, date_go, date_ms,

                                                  date_oracle, double, int32, int64, string. For each of the date types, the argument is a datetime layout string.

                                                  For the binary type, the argument can be one of: base32, base64, hex. All other types take an empty argument.

                                                  Only valid for CSV and TSV imports. e.g. zipcode.string(), thumbnail.binary(base64)

ingest options:

      –drop                                      drop collection before inserting documents

      –ignoreBlanks                              ignore fields with empty values in CSV and TSV

      –maintainInsertionOrder                    insert documents in the order of their appearance in the input source

  -j, –numInsertionWorkers=<number>              number of insert operations to run concurrently (defaults to 1) (default: 1)

      –stopOnError                               stop importing at first insert/upsert error

      –mode=[insert|upsert|merge]                insert: insert only. upsert: insert or replace existing documents. merge: insert or modify existing documents.

                                                  defaults to insert

      –upsertFields=<field>[,<field>]*           comma-separated fields for the query part when –mode is set to upsert or merge

      –writeConcern=<write-concern-specifier>    write concern options e.g. –writeConcern majority, –writeConcern ‘{w: 3, wtimeout: 500, fsync: true, j: true}’

      –bypassDocumentValidation                  bypass document validation

然后去试试

➜  backup_restore mongoimport -d dialog -c dialog –file mongodb_dialog_dialog.json

2018-06-29T14:15:57.110+0800    connected to: localhost

2018-06-29T14:16:00.066+0800    [#…………………..] dialog.dialog    47.5MB/583MB (8.1%)

2018-06-29T14:16:03.070+0800    [####………………..] dialog.dialog    99.0MB/583MB (17.0%)

2018-06-29T14:16:06.079+0800    [######………………] dialog.dialog    157MB/583MB (26.9%)

2018-06-29T14:16:09.078+0800    [########…………….] dialog.dialog    212MB/583MB (36.3%)

2018-06-29T14:16:12.066+0800    [##########…………..] dialog.dialog    262MB/583MB (44.9%)

2018-06-29T14:16:15.097+0800    [#############………..] dialog.dialog    320MB/583MB (54.9%)

2018-06-29T14:16:18.066+0800    [###############………] dialog.dialog    371MB/583MB (63.5%)

2018-06-29T14:16:21.068+0800    [#################…….] dialog.dialog    417MB/583MB (71.4%)

2018-06-29T14:16:24.068+0800    [###################…..] dialog.dialog    474MB/583MB (81.2%)

2018-06-29T14:16:27.074+0800    [#####################…] dialog.dialog    521MB/583MB (89.3%)

2018-06-29T14:16:30.066+0800    [#######################.] dialog.dialog    571MB/583MB (97.9%)

2018-06-29T14:16:31.378+0800    [########################] dialog.dialog    583MB/583MB (100.0%)

2018-06-29T14:16:31.378+0800    imported 41714 documents

然后本地去看看数据如何:

先去用PyCharm中mongo插件看看

然后再去用MongoDB Compass看看:

【已解决】安装MongoDB Compass去图形化查看Mongo数据

【总结】

此处,就是去在线服务器中导出:

方式1: 本地连接在线host去导出

mongoexport -h IP –port PORT -u dialog -p pwd –authenticationDatabase dialog -d dialog -c dialog -o mongodb_dialog_dialog.json

方式2: 登录在线服务器去本地导出:

mongoexport –port PORT -u dialog -p pwd –authenticationDatabase dialog -d dialog -c dialog -o mongodb_dialog_dialog.json

后,然后去本地导入:

mongoimport -d dialog -c dialog –file mongodb_dialog_dialog.json

即可。

转载请注明:在路上 » 【已解决】同步导出在线Mongo中某个集合(表)到本地

发表我的评论
取消评论

表情

Hi,您需要填写昵称和邮箱!

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址
56 queries in 0.084 seconds, using 18.90MB memory