-
Notifications
You must be signed in to change notification settings - Fork 14
/
Copy pathNEWS
314 lines (213 loc) · 25.8 KB
/
NEWS
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
[03-10-2019] v0.0.12
Continuous Data Protection For GNU/Linux (cdpfgl) is also known as 'sauvegarde' project. It is a set of programs: 'cdpfglserver', 'cdpfglclient' and 'cdpfglrestore' as of now. These programs will save your files in a live continuous way that is to say while they are written to disks. One interesting thing is that the server 'cdpfglserver' is stateless and achieves deduplication at the block level (optionally with adaptive block size). As a result it does not use much memory and may run on small machines (for instance I ran one cdpfglserver on a 1Gb bananapi).
This is v0.0.12 release of this project and one major feature was added that makes that version not backward compatible. Please let me know that you use this project by sending me an email so that I'll take backward compatibility into account.
* New feature:
* Now the client compresses (or not) the data and sends it to the server. The server stores data as they are transmitted (compressed or not).
* -z, --compression command line options added to cdpfglclient and compression-type=x where x can be 0 (none) or 1 (zlib) option to client configuration file.
* Improved parts:
* Code refactoring and cleaning improved the code - Expect more code refactoring in the future.
You are encouraged to contribute to this project - by example by saying to the author that you are using it. In order to do so you can open issues and pull requests on github. The project uses github's facilities such as milestones, issue and projects. Milestones are opened until v0.0.16 and some enhancement issues have been opened too.
Contributors to this version:
* Olivier Delhomme
Links:
* Sauvegarde release v0.0.12: http://cdpfgl.delhomme.org/download/releases/cdpfgl-0.0.12.tar.xz
* Debian's jessie packages: http://cdpfgl.delhomme.org/download/packages/debian/0.0.12/
* Sauvegarde's project: https://github.com/dupgit/sauvegarde
* Sauvegarde's gitter: https://gitter.im/dupgit/sauvegarde
* Sauvegarde's openhub: https://openhub.net/p/sauvegarde
* travis continuous integration status: https://travis-ci.org/dupgit/sauvegarde
* sauvegarde's frescode: http://freshcode.club/projects/sauvegarde
[10-09-2017] v0.0.11
Continuous Data Protection For GNU/Linux (cdpfgl) is also known as 'sauvegarde' project. It is a set of programs: 'cdpfglserver', 'cdpfglclient' and 'cdpfglrestore' as of now. These programs will save your files in a live continuous way that is to say while they are written to disks. One interesting thing is that the server 'cdpfglserver' is stateless and achieves deduplication at the block level (optionally with adaptive block size). As a result it does not use much memory and may run on small machines (for instance I run one cdpfglserver on a 1Gb bananapi).
This is v0.0.11 release of this project and some features were added, some were improved and a bug corrected:
* New features:
* -n, --hostname option added to cdpfglrestore program in order to restore a file from a host that did not owned the file originally.
* messages answering a .json request are now json formatted.
* url /Stats.json is now answering a json string giving some basic server usage statistics.
* cdpfglserver now prints its listening port when in debug mode.
* Improved parts:
* examples are now included in man pages.
* now cdpfglclient does local database versioning to ease future migrations or changes
* code refactoring and cleaning has been done.
* Bug corrected
* commit 375003985549 corrected a bug that made the url statistics counted twice.
You are encouraged to contribute to this project. In order to do so you can open issues and pull requests on github. The project uses github's facilities such as milestones, issue and projects. Milestones are opened until v0.0.16 and some enhancement issues have been opened too.
Contributors to this version:
* Olivier Delhomme
Links:
* Sauvegarde release v0.0.11: http://cdpfgl.delhomme.org/download/releases/cdpfgl-0.0.11.tar.xz
* Debian's jessie packages: http://cdpfgl.delhomme.org/download/packages/debian/0.0.11/
* Sauvegarde's project: https://github.com/dupgit/sauvegarde
* Sauvegarde's gitter: https://gitter.im/dupgit/sauvegarde
* Sauvegarde's openhub: https://openhub.net/p/sauvegarde
* travis continuous integration status: https://travis-ci.org/dupgit/sauvegarde
* sauvegarde's frescode: http://freshcode.club/projects/sauvegarde
* voidlinux distribution: http://www.voidlinux.eu/
[29-11-2016] v0.0.10
Continuous Data Protection For GNU/Linux (cdpfgl) is also known as 'sauvegarde' project. It is a set of programs: 'cdpfglserver', 'cdpfglclient' and 'cdpfglrestore' as of now. These programs will save your files in a live continuous way that is to say while they are written to disks. One interesting thing is that the server 'cdpfglserver' is stateless and achieves deduplication at the block level (optionally with adaptive block size). As a result it does not use much memory and may run on small machines (for instance I run one cdpfglserver on a 1Gb bananapi).
This is v0.0.10 release of this project and some features were added and some small bugs corrected:
* Man pages were added for cdpfglclient, cdpfglserver and cdpfglrestore
* Code refactoring has been done:
* By simplifying some command line related parts
* By putting some generic code into libcdpfgl library
* New options has been added to cdpfglrestore program:
* -f, --all-files now restores all selected files (in conjunction with -r option)
* -g, --latest option selects only the latest version of each selected file
* -P, --parents option restores files with their full path (creates directories if needed).
* New option has been added to cdpfglclient program:
* -n, --no-scan now deactivates the initial scan of every directory to be saved when launching cdpfglclient. This option is a boolean value named no-scan in cdpfglclient configuration file.
* libcdpfgl is now built more cleanly and is also used in 'content-define-cut' project
Contributors to this version:
* Olivier Delhomme
Links:
* Sauvegarde's project: https://github.com/dupgit/sauvegarde
* Sauvegarde's gitter: https://gitter.im/dupgit/sauvegarde
* Sauvegarde's openhub: https://openhub.net/p/sauvegarde
* Sauvegarde release v0.0.10: http://cdpfgl.delhomme.org/download/releases/cdpfgl-0.0.10.tar.xz
* travis continuous integration status: https://travis-ci.org/dupgit/sauvegarde
* sauvegarde's frescode: http://freshcode.club/projects/sauvegarde
* voidlinux distribution: http://www.voidlinux.eu/
[24-08-2016] v0.0.9
'sauvegarde' project is also known as Continuous Data Protection For GNU/Linux (cdpfgl). It is a set of programs: 'cdpfglserver', 'cdpfglclient' and 'cdpfglrestore' as of now. They will save your files in a live continuous way that is to say while they are written to disks. One interesting thing is that the server 'cdpfglserver' is stateless and achieves deduplication at the block level (optionally with adaptive block size). As a result it does not use much memory and may run on small machines (for instance I run one cdpfglserver on a 1Gb bananapi).
This is v0.0.9 release of this project and some features were added and some major bugs corrected:
* Documentation has been reviewed and improved,
* Some code refactoring was done in some parts of the programs,
* Corrected bugs that crashed the programs (cdpfglserver or cdpfglclient) when a file was containing a ',' or a '\n' character. Correction of this bug affects the way file names and link names are stored by file_backend of server program. Thus older version of sauvegarde will not work with a flat meta file from 0.0.9 and 0.0.9 will not work with a flat meta file from 0.0.8. Getting an older flat meta file to work for 0.0.9 is a matter of base64 encoding file name and link name. Upon demand I can code a tool for this,
* New way of trapping signals in client now make it able to close client program as cleanly as we can especially the local sqlite database connection,
* New url /Data/Hash_Array.json has been added in cdpfglserver. It answers GET request with an extra header X-Get-Hash-Array that must contain a comma separated base64 encoded hash list,
* New option -e, --all-versions is now available in cdpfglrestore to list or restore all versions of a file. Restoring all versions of a file prepends each filename with it's backup date (it adds a number if files have the same date),
* 1110+ commits reached !
Contributors to this version:
* Olivier Delhomme
Links:
* Sauvegarde's project: https://github.com/dupgit/sauvegarde
* Sauvegarde's gitter: https://gitter.im/dupgit/sauvegarde
* Sauvegarde's openhub: https://openhub.net/p/sauvegarde
* Sauvegarde release v0.0.9: http://cdpfgl.delhomme.org/download/releases/cdpfgl-0.0.9.tar.xz
* travis continuous integration status: https://travis-ci.org/dupgit/sauvegarde
* sauvegarde's frescode: http://freshcode.club/projects/sauvegarde
* voidlinux distribution: http://www.voidlinux.eu/
[03-01-2016] v0.0.7
'sauvegarde' project is also known as Continuous Data Protection For GNU/Linux (cdpfgl). It is a set of programs: 'cdpfglserver', 'cdpfglclient' and 'cdpfglrestore' as of now. They will saves your files in a live continuous way that is to say while they are written to disks. One interesting thing is that the server 'cdpfglserver' is stateless and achieves deduplication at the block level. As a result it does not use much memory and may run on small machines (for instance I run one cdpfglserver on a 1Gb bananapi).
This is v0.0.7 release of this project and some major features were added:
* The project is still named 'sauvegarde' in github but I will name it Continuous Data Protection For GNU/Linux (cdpfgl) as much as I can. All the programs are now using cdpfgl acronym (even the libsauvegarde as been renamed libcdpfgl). An effort as been made in the source code to track down old french names such as 'serveur', 'restaure' and so on.
* In the same way I renamed 'Serveur' section into 'Server' section in configuration files. So v0.0.6 or older configuration files are not compatible with v0.0.7 and one may have to change this manually.
* Ability to exclude some files by extension or path. This adds a new file configuration option named 'exclude-list=' in [Client] section. It takes some basic regular expressions such as those given in example in 'client.conf' file. It also adds a new cdpfglclient command line option called '-x' or '--exclude'.
* Client is threaded and now uses at least 3 threads (one is used only to uncache cached buffers when the server comes alive again).
* Caching mechanism in client in case the server is unreachable. This is achieved by using tables in a the client's local sqlite database. It breaks compatibility form older versions ie v0.0.6 database is not usable as is with v0.0.7 (I'll do a migration script upon request).
* Change GSList hash_data_list from meta_data_t structure to a GList
structure that allows deleting elements while walking through it
at 0(1) cost.
* I also corrected some bugs as they were found.
* packaging directory now contains stuff for packaging for the distributions (voidlinux and debian as of now).
* dockerfiles directory contains Dockerfiles to build the whole project in different distributions (centos and voidlinux) in a light way.
While working on this version a performance problem was found. It was linked to a libmicrohttpd bug that was shrinking a buffer and did never grow it again leading to a very small buffer. I managed to make a small set of programs that demonstrated the bug and Christian Grothoff fixed it in minutes. Thank you very much Christian !
Contributors to this version:
* Olivier Delhomme
* Pierre Bourgin
Links:
* Sauvegarde's project: https://github.com/dupgit/sauvegarde
* Sauvegarde's openhub: https://openhub.net/p/sauvegarde
* Sauvegarde release v0.0.7: http://cdpfgl.delhomme.org/download/releases/cdpfgl-0.0.7.tar.xz
* travis continuous integration status: https://travis-ci.org/dupgit/sauvegarde
* sauvegarde's frescode: http://freshcode.club/projects/sauvegarde
* voidlinux distribution: http://www.voidlinux.eu/
[02-11-2015] v0.0.6
'sauvegarde' is a set of tools ('cdpfglserver', 'cdpfglclient' and 'cdpfglrestore' - as of now) that saves your files in a live continuous way that is to say while they are written to disks. One interesting thing is that the server (now named 'cdpfglserver') is stateless and achieves deduplication (at the block level). As a result it does not use much memory and may run on small machines (on a virtual machine for instance).
This is v0.0.6 release of this project and some major features were added:
* Upon comments made by Pierre Bourgin when trying to package sauvegarde's project for voidlinux distribution the program names changed and are now cdpfglserver, cdpfglclient and cdpfglrestore. 'cdpfgl' stands for 'Continuous Data Protection For Gnu/Linux'.
* Now using cdpfglrestore with the -w (--where) option one can restore a file to some specific directory specified along with the option.
* cdpfglclient has a new mode to calculate hashs. This mode is called 'adaptive blocksize' it can be invoked with -a 1 (--adaptive=1) command line option or by using 'adaptive=true' directive in 'Client' section of client.conf configuration file. This option allows client to calculate hash with an adaptive blocksize that depends of the size of the file. It works by steps. File whose size is under 32768 bytes are hashed with a 512 bytes blocksize, files under 262144 bytes with a 2048 bytes blocksize and so on until files whose size is greater than 134217728 that are hashed with a 262144 bytes blocksize. It is believed that doing so, deduplication will hit a higher rate. The counter part is that cdpfglclient program is slower for small files.
* -s (--buffersize) option has been added to cdpfglclient program in order to let one choose the cache that cdpfglclient may use before sending data to cdpfglserver. This option has no effect when the adaptive blocksize option has been chosen has the program will adapt this buffersize roughly to each file.
* This release also contains many bugfixes and memory leakage fixes.
* Memory allocation strategy has been reviewed at some points: when it's possible, avoid allocating memory at all, when we must allocate see if g_malloc() is usable else use g_malloc0() which is 1000 times slower than g_malloc.
Contributors to this version:
* Olivier Delhomme
Links:
* Sauvegarde's project: https://github.com/dupgit/sauvegarde
* Sauvegarde's openhub: https://openhub.net/p/sauvegarde
* Sauvegarde release v0.0.6: http://src.delhomme.org/download/sauvegarde/releases/sauvegarde-0.0.6.tar.xz
* travis continuous integration status: https://travis-ci.org/dupgit/sauvegarde
* sauvegarde's frescode: http://freshcode.club/projects/sauvegarde
* voidlinux distribution: http://www.voidlinux.eu/
[04-10-2015] v0.0.5
'sauvegarde' is a set of tools ('serveur', 'client' and 'restaure' - as of now) that saves your files in a live continuous way that is to say while they are written to disks. One interesting thing is that the server (named 'serveur') is stateless and achieves deduplication (at the block level). As a result it does not use much memory and may run on small machines (on a virtual machine for instance).
This is v0.0.5 release of this project. Some improvements were made as all what was expected in the roadmap (and more) has been coded:
* fanotify's code has been reviewed a bit and a thread has been created to process (if possible) files. It allows to begin file change notification very early, before the end of the directory carving.
* Files that are 128 MB or more are processed differently to avoid having them completely in memory. The problem for now is that it does a naive transfer with the blocs: it sends everything thus it may waste network bandwidth and serveur and client CPU and IO. There is a simple solution to this that I will code in a future release (planned for v0.0.8).
* New test script used in travis-ci to tests things a bit and may be avoid basic problems.
* New Icon in pixmap directory available from 16x16 to 512x512 in CC-BY-SA license. The image is used in publications (there is no GUI as of now).
* Begun a user manual for sauvegarde's project in directory 'manual'. One may found development documentation in 'docs' directory.
* Added libsauvegarde.pc to allow one to use this library in an other project.
* Corrected 2 major mistakes that prevented client to be efficient when processing files (to avoid processing them when already processed). As a result client in v0.0.5 is slower than in v0.0.4 but saves a lot of IO and CPU on its running machine.
Contributors to this version:
* Olivier Delhomme
Links:
* Sauvegarde's project: https://github.com/dupgit/sauvegarde
* Sauvegarde's openhub: https://openhub.net/p/sauvegarde
* Sauvegarde release v0.0.5: http://src.delhomme.org/download/sauvegarde/releases/sauvegarde-0.0.5.tar.xz
* travis continuous integration status: https://travis-ci.org/dupgit/sauvegarde
[06-09-2015] v0.0.4
'sauvegarde' is a set of tools ('serveur', 'client' and 'restaure' - as of now) that saves your files in a live continuous way that is to say while they are written to disks. One interesting thing is that the server (named 'serveur') is stateless and achieves deduplication (block level). As a result it does not use much memory and may run on small machines (on a virtual machine for instance).
This is the fourth release of this project. Some improvements were made. Let's see what's new in version v0.0.4:
* There is now a new server url to post a bunch of hashs and associated data (/Data_Array.json). The JSON string expected must contain an array named 'data_array'. Each object of this array must contain the three fields 'hash', 'data' and 'size'. Fields 'hash' and 'size' must be base64 encoded. This has the effect of buffering the communication a bit. My tests on my single computer showed that the gain of speed is at least 4 times.
* When sending the hashs of a file in it's meta data the server answers the hashs that it needs (unknown to him). But if the file has several times the same hash that is unknown the server was answering as many times to send this hash. Now, with v0.0.4 the answer has only unique hashs, avoiding the client to send several times the same block.
* file_backend has now a configuration section into the 'serveur.conf' file named '[file_backend]'. Two options can be configured. An option named 'file-directory' that tells the backend where to put it's files and 'dir-level' that tells the backend the number of level we want to store data. It's default value is 2 (it means that serveur will create 65536 directories). The value is limited to a maximum of 5 (ie 256^5 = 1 099 511 627 776 directories!). Keep in mind that creating the directories may last a long time if you choose a high value (It will only be done once) and also that a directory may take some space (on ext4 a level 2 takes 256 Mb but level 3 takes 64 Gb !).
* sauvegarde is now fully translated in French and is ready for other translations (it is based on .po files).
* Sébastien Tricaud patch was merged in this version adding the ability to catch SIGINT and clean the memory before exiting avoiding a memory leak.
* A manual has been created and is waiting for contributions at http://write.flossmanuals.net/sauvegarde-manual/_info/.
* TODO file has been reworked and contains new ideas that I might put in the roadmap.
Contributors to this version:
* Olivier Delhomme
* Sébastien Tricaud
Links:
* Sauvegarde's project: https://github.com/dupgit/sauvegarde
* Sauvegarde's manual (still to be created): http://write.flossmanuals.net/sauvegarde-manual/_info/
* Sauvegarde's openhub: https://openhub.net/p/sauvegarde
* Sauvegarde release v0.0.4: http://src.delhomme.org/download/sauvegarde/releases/sauvegarde-0.0.4.tar.xz
* travis continuous integration status: https://travis-ci.org/dupgit/sauvegarde
[21-08-2015] v0.0.3
'sauvegarde' is a set of tools ('serveur', 'client' and 'restaure' - as of now) that saves your files in a live continuous way that is to say while they are written to disks.
New features in version v0.0.3:
* links are now saved and can be restored
* A new test directory comes with the project where we might put some files, directories or links to test, improve and avoid regression on the project.
There is a lot of things to do and sauvegarde's project is seeking help in all range of the project (including testing, promoting, coding, translating, creating documentations...).
Links:
* Sauvegarde's project: https://github.com/dupgit/sauvegarde
* Sauvegarde's openhub: https://openhub.net/p/sauvegarde
* Sauvegarde release v0.0.3: http://src.delhomme.org/download/sauvegarde/releases/sauvegarde-0.0.3.tar.xz
* travis continuous integration status: https://travis-ci.org/dupgit/sauvegarde
[15-08-2015] v0.0.2
'sauvegarde' is a set of tools ('serveur', 'client' and 'restaure' - as of now) that saves your files in a live continuous way that is to say while they are written to disks. In this version 'client' program has been reworked and may be 75 % of it's code has changed.
New features in version v0.0.2:
* client has been reworked and there is no more threads. As a result, the functions newly written are not threadable as is.
* file and time requests are base64 encoded into the URL so that it can manage space without errors
* client save written files in a live an continuous way
* time measurement is now possible (for instance, on my computer client uses 1.5ms to know if a file is in it's cache or not)
* restore can restore a file at a specific date (you'll have to have the right date or it'll restore the latest one in the retrieved list)
There is still a lot of work to do and one might want to look at the list of my first post at http://blog.delhomme.org/index.php?post/2015/07/08/Sauvegarde-project-first-usable-version or see the TODO file or the Roadmap section of README file or even look into the code (or doxygen documentation) for @todo tags. Any help, contributions, remarks, articles or encouragements are very welcomed.
The whole project is known to compile on x86_64 and arm7l architecture and on Debian Jessie, Ubuntu 12.04 and Centos 7 (sometimes at the expense of compiling and installing more recent dependencies - which is pretty straight (look for .travis.yml in the source code)).
Links:
* Sauvegarde's project: https://github.com/dupgit/sauvegarde
* Sauvegarde release v0.0.2: http://src.delhomme.org/download/sauvegarde/releases/sauvegarde-0.0.2.tar.xz
* travis continuous integration status: https://travis-ci.org/dupgit/sauvegarde
[09-07-2015] v0.0.1
Since February 2014 I'm programming, when I have some time, something that may save my files live. The primary goal is to save files while they are being created or modified. It is still a goal to reach even if the first usable version is out today ! When I say usable I mean that one can backup files and restore them. It needs a lot of improvements and a lot of fine code review to track down the numerous memory leakages and bugs that are in this early code (Yes contributions are welcome :).
Naming is usually a very difficult task for a project and as I remarked that successful projects have simple names - like "Word" and "Windows" for instance (yes I know this is not free software) I named my project "sauvegarde" (which means backup in French). This project is composed of 3 programs right now, the one that saves your file is named "client" and is talking to the server (named "serveur" in French). The third one can restore files (also by talking to the server) and is named "restaure" (restore in French).
The whole project is known to compile on x86_64 and arm7l architecture and on Debian Jessie, Ubuntu 12.04 and Centos 7 (sometimes at the expense of compiling and installing more recent dependencies - which is pretty straight (look for .travis.yml in the source code)).
With this version first you have to setup a server and have serveur running on it. Then you have to run client each time you need to backup things (live backup will come very soon - even if linux kernel FAN notification's interface lacks some notifiers). When you want to restore a file you have to use "restaure" program (server needs to be up and accessible). At the moment you can only restore the latest version of a file.
The server has an interface to connect different backends. The first one is the flat file backend. One may program some database backend (postgresql, sqlite, mariadb...), or a different storage backend like Ceph for instance. All the data is saved in a deduplicated way and the network traffic is minimised. "client", "serveur" and "restore" talks in json language to each others.
There is still a lot of things to do such as (in no specific order):
* Make live backup possible
* Having the ability to restore any version of a file
* Having the ability to restore a group of files
* Having the ability to restore a directory (may need to patch linux's kernel FAN interface)
* Saving symbolic links
* Having some security for transfers (for instance https)
* Having some security for authentication of programs each others
* Adding server backends
* Internationalization
If you test this program, if you compile it under a new architecture/system, if you have any question or want to report a bug please feel free to use either github's interface or your favourite email program to email me (my email address is in the source code or in AUTHORS file).
Links:
* Sauvegarde's project: https://github.com/dupgit/sauvegarde
* travis continuous integration status: https://travis-ci.org/dupgit/sauvegarde