Added a backup tool to import/export store data #35

marcjamot · 2015-09-29T09:37:15Z

Added a backup tool in order to import and export data for the OpenBazaar server.
It has four public methods which handles backup/restore of files and export/import of data from the database with connections to the API for the client. All the methods base their work directory from OpenBazaar directory, usually "~/OpenBazaar".

backupFiles takes an output file name/path and archives all the content of the OpenBazaar directory to that file.

exportDatabase takes a list of table names and exports their data as CSV files to "OpenBazaar/backup/" directory. It also takes a boolean flag to remove previous files before generation.

restoreFiles imports the files from the given archive to the OpenBazaar directory, overwriting previous files. It also takes a boolean flag to remove previous CSV files since the importDatabase method imports all CSV files in backup directory.

importDatabase imports all CSV files from backup folder to the table, used to import data. It also takes a boolean flag to replace data instead of appending it.

gubatron · 2015-09-29T11:33:41Z

might want to make all your methods with pythonic_names, in one module you did it fine but on the other one you used camelCase (I personally love camelCase better, but this is a python project and I bet you will get "yelled" at for that at some point)

gubatron · 2015-09-29T11:35:27Z

backupTool.py

+import tarfile
+import time
+
+_TABLES = {


think there might be a way to initialize this programatically out of the database schema? this seems like a maintenance hell.

hoffmabc · 2015-09-29T11:36:44Z

Thanks for the submission. Will review this morning. As @gubatron mentioned we want to have consistency on code and as you can see the build failed. Just make sure it passes the pylint checks. They include the naming conventions.

gubatron · 2015-09-29T11:38:02Z

backupTool.py

+def _exportDatabaseToCsv(tablesAndColumns):
+    """Reads the database for all given tables and stores them as CSV files."""
+    dbFile = _getDatabase()
+    result = None


result looks like an unused variable that you forgot to clean up.

marcjamot · 2015-09-29T12:57:30Z

Yeah, I'll run it through pylint to make sure it goes well. Also the return True part makes sense. For the import of tables, does it make sense to do as now where it imports all the CSV files or should it be changed to also take a list of tables as export does? Seems the latter makes more sense when it's part of an API.

cpacia · 2015-10-01T17:45:44Z

Thanks for this. I will review it this week.

gubatron · 2015-10-01T18:11:09Z

@PLANKT :

Few more questions:

Does the database have indexes you might want to re-create when importing?
Instead of doing the exporting/backup manually, did you have the chance to take a look at sqlite's backup api? (I'm concerned about the maintenance issues this will have, specially the part where you sort of recreate the schema at the beginning on that _TABLES dict. it will be a never ending bug fountain forgetting to come back here and adding/removing/renaming fields and tables, if you intend to do the backups with the functions coded in the patch, you should try and form that _TABLES dict programatically out of the DB schema.)

cpacia · 2015-10-03T17:58:34Z

I haven't had the time to test it yet, but the one comment I would make is I would put backuptool.py in the db subfolder.

marcjamot · 2015-10-12T22:29:48Z

Regarding indexes and table structure, as mentioned it would be convenient for the backup tool to simply handle states, then it won't matter if the tables change in any way as long as the backup is between compatible versions. I looked into two solutions:

Backup the file. By locking the database file and doing a copy of the database file, all the data is stored and can be restored for a given time. This blocks any database access during the access and can be done with the default sqlite package.

Sqlite's official backup API seems to be missing from the Python sqlite package, there is however a module that I found have been recommended from different sites.

husio/python-sqlite3-backup (https://github.com/husio/python-sqlite3-backup). The backup can be done from file or memory to file or memory, so no lock for the whole database is needed. Done between two connections as simply as "sqlitebck.copy(conn, conn2)".

In both scenarios the single table backup functionality would be removed, it makes more sense to backup/restore between states of the whole database, and if single table export is wanted it could be an additional feature for datastore to prevent potential missed bugs when changing table structures.

gubatron · 2015-10-14T18:43:30Z

q: what happened to the previous backup logic? I remember coding somethign which would basically make a copy of the entire db folder and .tgz it.

marcjamot · 2015-10-16T08:05:54Z

I changed the tool to only include two methods to simplify and do a state backup. Also moved it to db folder. The "backup" locks the database and does a full backup of the OpenBazaar data directory for the user (except the backup folder where it places the backups) and the "restore" restores the backup.

…into develop Conflicts: api/restapi.py

gasull · 2015-12-06T16:11:25Z

db/tests/test_backuptool.py

+TEST_TAR = 'test.tar.gz'
+TEST_FILE_1 = 'test.txt'
+TEST_FILE_2 = TEST_FOLDER + os.sep + 'test2.txt'
+


PEP8-nitpicking: Please add an extra blank line here, before the class definition.

cpacia · 2015-12-11T20:57:20Z

This is something I'd like to revisit once the database schema is set for this version. Also need to make sure the json contracts get backed up.

jonathancross · 2016-12-18T19:49:23Z

Hi, it has been more than a year since @cpacia's last comment. Should this PR be closed or is there a small chance it could be improved and merged?

gubatron · 2016-12-18T20:26:42Z

try rebasing and see if this even makes sense anymore. I'd close, this will probably be irrelevant very soon now that the server for OB2 will be made in go

hoffmabc · 2017-06-18T12:23:33Z

Thanks for the bump @gasull. I think we are pretty much only making minor changes or bug fixes as a priority due to 2.0.

@cpacia do you want to review this again and perhaps if it makes sense we can put it in the next release?

hauxir and others added 9 commits September 15, 2015 09:53

temp fix for OB server

3655048

test form

e812d26

Add base functionality for backup tool python code

4a69399

Remove print in code

cd97a39

Updated backup to use generated OpenBazaar folder

ec1caf4

Add API support for backup tool

8b01ed3

Add backup test html

1e8a6ec

Separate archive from database export

d18ca0a

Added a backup tool to backup your store data

e4fe5fd

marcjamot force-pushed the develop branch 2 times, most recently from d5fa98f to 2fea134 Compare September 29, 2015 10:08

gubatron reviewed Sep 29, 2015
View reviewed changes

marcjamot force-pushed the develop branch from 2fea134 to 9a948f4 Compare October 1, 2015 14:14

marcjamot force-pushed the develop branch 3 times, most recently from fecd9da to ca2e123 Compare October 16, 2015 07:57

hauxir force-pushed the develop branch from 1a8e0a0 to e4fe5fd Compare November 10, 2015 13:08

hauxir and others added 2 commits November 13, 2015 15:32

test form

4469bf6

Add base functionality for backup tool python code

4949aa6

marcjamot added 6 commits November 13, 2015 15:32

Remove print in code

e64a0e2

Updated backup to use generated OpenBazaar folder

ee4bd6c

Add backup test html

54cd325

Add test for backup tool

1e2daef

Rebase master

e7a5246

Merge branch 'develop' of https://github.com/hauxir/OpenBazaar-Server …

14053e8

…into develop Conflicts: api/restapi.py

gasull reviewed Dec 6, 2015
View reviewed changes

cpacia force-pushed the master branch from 02b52f7 to 0780c1d Compare January 25, 2016 20:39

cpacia force-pushed the master branch 4 times, most recently from 6bf74b9 to 4aac065 Compare March 4, 2016 17:06

drwasho added the Enhancement/suggestion label Mar 28, 2016

drwasho added this to the Future version milestone Mar 28, 2016

cpacia force-pushed the master branch from 2165dc1 to b8e39a3 Compare April 11, 2016 17:49

drwasho removed this from the Future version milestone Apr 20, 2016

gasull mentioned this pull request Jun 18, 2017

Pr/35 #527

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added a backup tool to import/export store data #35

Added a backup tool to import/export store data #35

marcjamot commented Sep 29, 2015

gubatron commented Sep 29, 2015

gubatron Sep 29, 2015

hoffmabc commented Sep 29, 2015

gubatron Sep 29, 2015

marcjamot commented Sep 29, 2015

cpacia commented Oct 1, 2015

gubatron commented Oct 1, 2015

cpacia commented Oct 3, 2015

marcjamot commented Oct 12, 2015

gubatron commented Oct 14, 2015

marcjamot commented Oct 16, 2015

gasull Dec 6, 2015

cpacia commented Dec 11, 2015

jonathancross commented Dec 18, 2016

gubatron commented Dec 18, 2016

hoffmabc commented Jun 18, 2017

Added a backup tool to import/export store data #35

Are you sure you want to change the base?

Added a backup tool to import/export store data #35

Conversation

marcjamot commented Sep 29, 2015

gubatron commented Sep 29, 2015

gubatron Sep 29, 2015

Choose a reason for hiding this comment

hoffmabc commented Sep 29, 2015

gubatron Sep 29, 2015

Choose a reason for hiding this comment

marcjamot commented Sep 29, 2015

cpacia commented Oct 1, 2015

gubatron commented Oct 1, 2015

cpacia commented Oct 3, 2015

marcjamot commented Oct 12, 2015

gubatron commented Oct 14, 2015

marcjamot commented Oct 16, 2015

gasull Dec 6, 2015

Choose a reason for hiding this comment

cpacia commented Dec 11, 2015

jonathancross commented Dec 18, 2016

gubatron commented Dec 18, 2016

hoffmabc commented Jun 18, 2017