From a223e22982f0ca197a643a57b93cbcc066bc0b6d Mon Sep 17 00:00:00 2001 From: Sam Silverberg Date: Sat, 28 Nov 2015 18:07:03 -0800 Subject: [PATCH] fixed java references in shell scripts --- README.md | 266 +++++------ .../deb/usr/share/sdfs/changes.txt | 25 + install-packages/deb/usr/share/sdfs/mkfs.sdfs | 2 +- install-packages/deb/usr/share/sdfs/odvol.cli | 2 +- .../deb/usr/share/sdfs/odvol.make | 2 +- .../deb/usr/share/sdfs/odvol.start | 43 +- .../deb/usr/share/sdfs/readme.txt | 452 +++++++++--------- install-packages/deb/usr/share/sdfs/sdfscli | 2 +- .../deb/usr/share/sdfs/sdfsreplicate | 2 +- install-packages/windows/readme-win32.txt | 8 +- src/readme-win32.txt | 139 +----- src/readme.txt | 12 +- 12 files changed, 461 insertions(+), 494 deletions(-) diff --git a/README.md b/README.md index a093c2593..671433fc0 100644 --- a/README.md +++ b/README.md @@ -1,140 +1,128 @@ # sdfs +What is this? + + A clustered deduplicated file system. + Requirements System Requirements - 1. x64 Linux Distribution. The application was tested and developed on ubuntu 14.0.4 and CentOS 7 - 2. At least 4 GB of RAM - 3. Java 7 - available at https://jdk7.dev.java.net/ + 1. x64 Linux Distribution. The application was tested and developed on ubuntu 14.0.4 and CentOS 7 + 2. At least 4 GB of RAM + 3. Java 8 - available at https://jdk8.dev.java.net/ + 4. JSVC (http://commons.apache.org/proper/commons-daemon/jsvc.html) + Optional Packages - NFS Kernel Server - LIO - for ISCSI support - Installation + * NFS Kernel Server + * LIO - for ISCSI support -Ubuntu/Debian (Ubuntu 15.10) +Installation - Step 1: Install Java JRE - sudo apt-get install openjdk-7-jre-headless +Ubuntu/Debian (Ubuntu 14.04+) - Step 2: Install SDFS File System - wget http://www.opendedup.org/downloads/sdfs-2.0.11_amd64.deb - sudo dpkg -i sdfs-2.0.11_amd64.deb + Step 1: Download the latest sdfs version + wget http://opendedup.org/downloads/sdfs-latest.deb + + Step 2: Install sdfs and dependencies + sudo apt-get install fuse libfuse2 ssh openssh-server jsvc libxml2-utils + sudo dpkg -i sdfs-latest.deb + Step 3: Change the maximum number of open files allowed - sudo su echo "* hardnofile 65535" >> /etc/security/limits.conf echo "* soft nofile 65535" >> /etc/security/limits.conf exit - Step 4: Log Out and Proceed to Initialization Instructions - -CentOS/RedHat (Centos 6.5) - - Step 1: Log in as root + Step 5: Log Out and Proceed to Initialization Instructions - Step 2: Install Java JRE +CentOS/RedHat (Centos 7.0+) - yum install java-1.7.0-openjdk - - Step 3: Install the SDFS File System - - wget http://www.opendedup.org/downloads/SDFS-2.0.11-2.x86_64.rpm - rpm -iv SDFS-2.0.11-2.x86_64.rpm - Step 4: Change the maximum number of open files allowed + Step 1: Download the latest sdfs version + wget http://opendedup.org/downloads/sdfs-latest.rpm + Step 2: Install sdfs and dependencies + yum install jsvc libxml2 java-1.8.0-openjdk + rpm -iv --force sdfs-latest.rpm + + Step 3: Change the maximum number of open files allowed echo "* hardnofile 65535" >> /etc/security/limits.conf echo "* soft nofile 65535" >> /etc/security/limits.conf exit - Step 5: Disable the IPTables firewall - + Step 5: Log Out and Proceed to Initialization Instructions + + Step 6: Disable the IPTables firewall + service iptables save service iptables stop chkconfig iptables off - Step 4: Log Out and Proceed to Initialization Instructions + Step 7: Log Out and Proceed to Initialization Instructions -Fixed Block Deduplication +Initialization Instructions for Standalone Volumes - Step 1: Log into the linux system as root or use sudo - Step 2: Create the SDFS Volume. This will create a volume with 256 GB of capacity using a 4K block size. + Step 1: Log into the linux system as root or use sudo + Step 2: Create the SDFS Volume. This will create a volume with 256 GB of capacity using a Variable block size. + **Local Storage** sudo mkfs.sdfs --volume-name=pool0 --volume-capacity=256GB + **AWS Storage** + sudo mkfs.sdfs --volume-name=pool0 --volume-capacity=1TB --aws-enabled true --cloud-access-key --cloud-secret-key --cloud-bucket-name + + **Azure Storage** + sudo mkfs.sdfs --volume-name=pool0 --volume-capacity=1TB --azure-enabled true --cloud-access-key --cloud-secret-key --cloud-bucket-name + + **Google Storage** + sudo mkfs.sdfs --volume-name=pool0 --volume-capacity=1TB --google-enabled true --cloud-access-key --cloud-secret-key --cloud-bucket-name + + Step 3: Create a mount point on the filesystem for the volume sudo mkdir /media/pool0 Step 4: Mount the Volume - sudo mount.sdfs pool0 /media/pool0/ & - - Step 5: Copy some data to the volume and make sure it all is there - - md5sum /etc/sdfs/pool0-volume-cfg.xml - cp /etc/sdfs/pool0-volume-cfg.xml /media/pool0 - md5sum /media/pool0/pool0-volume-cfg.xml - Step 6 (optional) : Unmount the volume - umount /media/pool0 - -Variable Block Deduplication - - Step 1: Log into the linux system as root or use sudo - - Step 2: Create the SDFS Volume. This will create a volume with 256 GB of capacity using a variable block size. - - sudo mkfs.sdfs --volume-name=pool0 --volume-capacity=256GB --hash-type=VARIABLE_MURMUR3 - - Step 3: Create a mount point on the filesystem for the volume - - sudo mkdir /media/pool0 + sudo mount -t sdfs pool0 /media/pool0/ - Step 4: Mount the Volume - - sudo mount.sdfs pool0 /media/pool0/ & + Set 5: Add the filesystem to fstab + pool0 /media/pool0 sdfs defaults 0 0 + - Step 5: Copy some data to the volume and make sure it all is there - md5sum /etc/sdfs/pool0-volume-cfg.xml - cp /etc/sdfs/pool0-volume-cfg.xml /media/pool0 - md5sum /media/pool0/pool0-volume-cfg.xml - Step 6 (optional) : Unmount the volume - umount /media/pool0 Initialization Instructions for A Multi-Node Configuration -In a mult-node cluster there are two components : - -The Dedup Storage Engine (DSE) - This is the server/service that store all unique blocks. For redundancy, you should have at least two of these. They can live on the same physical server as other components but this is not recommended for redundancy -The File System Service (FSS) - This is the server/service that mounts the sdfs volume. Multiple FSS services each serving its own volume can live in the same cluster. - -Prerequisits : - -Static IP address and unique name for all nodes: All nodes should have a static IP address to reduce network issues. -Disable host firewalls : IPTables should be disabled for testing and then enabled again if needed with open ports for multicast and udp. -Network Communication : Multicast communication is the default configuration. All nodes in the cluster will need to be able to talk to eachother using multicast unless the Jgroups configuration is modified.Multicast communication is usually not a problem on the same subnet but can be an issue between subnets or over wans -Fast Low Latency Network : SDFS will work within most networks but speed will suffer is network speeds low or network latency is high. 1Gb/s or above dedicated network is recommeded. -Update your /etc/sysctl.conf to improve network IO performance. Below are some recommeded options : - vm.swappiness=1 - net.core.rmem_max=254800000 - net.core.wmem_max=254800000 - net.core.rmem_default=254800000 - net.core.wmem_default=254800000 - net.core.optmem_max=25480000 - net.ipv4.tcp_timestamps=0 - net.ipv4.tcp_sack=0 - net.core.netdev_max_backlog=250000 - net.ipv4.tcp_mem=25480000 25480000 25480000 - net.ipv4.tcp_rmem=4096 87380 25480000 - net.ipv4.tcp_wmem=4096 65536 25480000 - net.ipv4.tcp_low_latency=1 - -Creating and Starting a DSE + In a mult-node cluster there are two components : + + * The Dedup Storage Engine (DSE) - This is the server/service that store all unique blocks. For redundancy, you should have at least two of these. They can live on the same physical server as other components but this is not recommended for redundancy + * The File System Service (FSS) - This is the server/service that mounts the sdfs volume. Multiple FSS services each serving its own volume can live in the same cluster. + Prerequisits : + + * Static IP address and unique name for all nodes: All nodes should have a static IP address to reduce network issues. + * Disable host firewalls : IPTables should be disabled for testing and then enabled again if needed with open ports for multicast and udp. + * Network Communication : Multicast communication is the default configuration. All nodes in the cluster will need to be able to talk to eachother using multicast unless the Jgroups configuration is modified.Multicast communication is usually not a problem on the same subnet but can be an issue between subnets or over wans + * Fast Low Latency Network : SDFS will work within most networks but speed will suffer is network speeds low or network latency is high. 1Gb/s or above dedicated network is recommeded. + * Update your /etc/sysctl.conf to improve network IO performance. Below are some recommeded options : + vm.swappiness=1 + net.core.rmem_max=254800000 + net.core.wmem_max=254800000 + net.core.rmem_default=254800000 + net.core.wmem_default=254800000 + net.core.optmem_max=25480000 + net.ipv4.tcp_timestamps=0 + net.ipv4.tcp_sack=0 + net.core.netdev_max_backlog=250000 + net.ipv4.tcp_mem=25480000 25480000 25480000 + net.ipv4.tcp_rmem=4096 87380 25480000 + net.ipv4.tcp_wmem=4096 65536 25480000 + net.ipv4.tcp_low_latency=1 + Creating and Starting a DSE The following steps will create 2 DSEs on server1 and server2 Step 1: Create a DSE on Server1 using a 4K block size and 200GB of capacity and cluster node id of "1" - + mkdse --dse-name=sdfs --dse-capacity=200GB --cluster-node-id=1 - + Step 2: Edit the /etc/sdfs/jgroups.cfg.xml and add the bind_addr attribute with Server1's IP address to the tag. tag. - + /sys/devices/virtual/net/$INTERFACE/bridge/multicast_querier - fi - -Testing multicast support on nodes in the cluster + By default KVM networking does not seem to allow guest to communicate over multicast. It also doesn't seem to work when bridging from a nic. From my reseach it looks like you have to setup a routed network from a KVM host and have all the guests on that shared network. In addition, you will want to enable multicast on the virtual nic that is shared by those guest. Here is is the udev code to make this happen. A reference to this issue is found here. + + # cat /etc/udev/rules.d/61-virbr-querier.rules + ACTION=="add", SUBSYSTEM=="net", RUN+="/etc/sysconfig/network-scripts/vnet_querier_enable" + + # cat /etc/sysconfig/network-scripts/vnet_querier_enable + #!/bin/sh + if [[ $INTERFACE == virbr* ]]; then + /bin/echo 1 > /sys/devices/virtual/net/$INTERFACE/bridge/multicast_querier + fi + + Testing multicast support on nodes in the cluster The jgroups protocol includes a nice tool to verify multicast is working on all nodes. Its an echo tool and sends messages from a sender to a reciever - + On the receiver run - - java -cp /usr/share/sdfs/lib/jgroups-3.4.1.Final.jar org.jgroups.tests.McastReceiverTest -mcast_addr 231.12.21.132 -port 45566 - + + java -cp /usr/share/sdfs/lib/jgroups-3.4.1.Final.jar org.jgroups.tests.McastReceiverTest -mcast_addr 231.12.21.132 -port 45566 + On the sender run - - java -cp /usr/share/sdfs/lib/jgroups-3.4.1.Final.jar org.jgroups.tests.McastSenderTest -mcast_addr 231.12.21.132 -port 45566 - - Once you have both sides running type a message on the sender and you should see it on the receiver after you press enter. You may also want to switch rolls to make sure multicast works both directions. - + + java -cp /usr/share/sdfs/lib/jgroups-3.4.1.Final.jar org.jgroups.tests.McastSenderTest -mcast_addr 231.12.21.132 -port 45566 + + Once you have both sides running type a message on the sender and you should see it on the receiver after you press enter. You may also want to switch rolls to make sure multicast works both directions. + take a look at http://docs.jboss.org/jbossas/docs/Clustering_Guide/4/html/ch07s07s11.html for more detail. -Further reading: + Further reading: -Take a look at the administration guide for more detail. http://www.opendedup.org/sdfs-20-administration-guide + Take a look at the administration guide for more detail. http://www.opendedup.org/sdfs-20-administration-guide -Ask for Help + Ask for Help -If you still need help check out the message board here https://groups.google.com/forum/#!forum/dedupfilesystem-sdfs-user-discuss + If you still need help check out the message board here https://groups.google.com/forum/#!forum/dedupfilesystem-sdfs-user-discuss \ No newline at end of file diff --git a/install-packages/deb/usr/share/sdfs/changes.txt b/install-packages/deb/usr/share/sdfs/changes.txt index fda04da7c..d0b4c6a47 100644 --- a/install-packages/deb/usr/share/sdfs/changes.txt +++ b/install-packages/deb/usr/share/sdfs/changes.txt @@ -1,3 +1,28 @@ +Version 3.0.1 + This is a major new release of SDFS with a rewrite of the storage subsystem. This release includes enhancements to the following elements + + Cloud Storage Enhancements : + * In-Line deduplication to a Cloud Storage Backend - SDFS can send all of its data to AWS, AZURE, or Google. 2.Performance Improvements - Compressed Multi-Threaded upload and download of data to the cloud + * Local Cache - SDFS will cache the most recently accessed data locally. This is configurable but set to 10 GB by Default + * Security - All data can be encrypted using AES-CBC 256 when sent to the cloud + * Throttling - Upload and download speeds can be throttled. + * Cloud Recovery/Replication - All local metadata is replicated to the cloud and can be recovered back to local volume. + * Glacier Support - Supports S3 Lifecycle policies and retrieving data from Glacier + * AWS Region Support - Supports all AWS Regions + * Amazon AIM Support + + Scalability Improvements: + * Tested to over 100TB of backend storage. + * Requires only 400MB of memory per TB of unique storage + * Unlimited Snapshots + * No compaction necessary sing orphaned data physically removed from underlying filesystem after garbage collection. + + General Enhancements : + * Mounts from fstab and using native linux mount commands (mount -t sdfs) + * Faster Volume Replication + * Local Encryption support - Uses AES-CBC 256 + * Faster Garbage Collection - 300% improvement + Version 2.0.11 Fixes : * Fixed copy of files with non-english characters diff --git a/install-packages/deb/usr/share/sdfs/mkfs.sdfs b/install-packages/deb/usr/share/sdfs/mkfs.sdfs index 04d7f29c3..b85cfa2c3 100755 --- a/install-packages/deb/usr/share/sdfs/mkfs.sdfs +++ b/install-packages/deb/usr/share/sdfs/mkfs.sdfs @@ -1,2 +1,2 @@ -java -Dfile.encoding=UTF-8 -classpath /usr/share/sdfs/lib/sdfs.jar:/usr/share/sdfs/lib/* org.opendedup.sdfs.VolumeConfigWriter "$@" +/usr/share/sdfs/bin/jre/bin/java -Dfile.encoding=UTF-8 -classpath /usr/share/sdfs/lib/sdfs.jar:/usr/share/sdfs/lib/* org.opendedup.sdfs.VolumeConfigWriter "$@" diff --git a/install-packages/deb/usr/share/sdfs/odvol.cli b/install-packages/deb/usr/share/sdfs/odvol.cli index b07c237d5..71c5174cb 100755 --- a/install-packages/deb/usr/share/sdfs/odvol.cli +++ b/install-packages/deb/usr/share/sdfs/odvol.cli @@ -1 +1 @@ -java -classpath /usr/share/sdfs/lib/sdfs.jar:/usr/share/sdfs/lib/* org.opendedup.sdfs.mgmt.cli.SDFSCmdline "$@" +/usr/share/sdfs/bin/jre/bin/java -classpath /usr/share/sdfs/lib/sdfs.jar:/usr/share/sdfs/lib/* org.opendedup.sdfs.mgmt.cli.SDFSCmdline "$@" diff --git a/install-packages/deb/usr/share/sdfs/odvol.make b/install-packages/deb/usr/share/sdfs/odvol.make index 5369a2e62..db4984e5e 100755 --- a/install-packages/deb/usr/share/sdfs/odvol.make +++ b/install-packages/deb/usr/share/sdfs/odvol.make @@ -1,2 +1,2 @@ -java -Dfile.encoding=UTF-8 -classpath /usr/share/sdfs/lib/sdfs.jar:/usr/share/sdfs/lib/* org.opendedup.sdfs.VolumeConfigWriter "$@" --io-safe-sync=true +/usr/share/sdfs/bin/jre/bin/java -Dfile.encoding=UTF-8 -classpath /usr/share/sdfs/lib/sdfs.jar:/usr/share/sdfs/lib/* org.opendedup.sdfs.VolumeConfigWriter "$@" --io-safe-sync=true diff --git a/install-packages/deb/usr/share/sdfs/odvol.start b/install-packages/deb/usr/share/sdfs/odvol.start index 9d1636adf..3735fb38a 100755 --- a/install-packages/deb/usr/share/sdfs/odvol.start +++ b/install-packages/deb/usr/share/sdfs/odvol.start @@ -1,3 +1,42 @@ export LD_LIBRARY_PATH=/usr/share/sdfs/bin/:$LD_LIBRARY_PATH -java -Djava.library.path=/usr/share/sdfs/bin/ -Dorg.apache.commons.logging.Log=fuse.logging.FuseLog\ - -Dfuse.logging.level=INFO -Xmx2g -Xms2g -server -XX:+DisableExplicitGC -XX:+UseParallelGC -XX:+UseParallelOldGC -XX:ParallelGCThreads=4 -XX:InitialSurvivorRatio=3 -XX:TargetSurvivorRatio=90 -Djava.awt.headless=$ -XX:+UseCompressedOops -classpath /usr/share/sdfs/lib/sdfs.jar:/usr/share/sdfs/lib/* org.opendedup.buse.sdfsdev.SDFSVolMgr "$@" +#!/bin/bash +modprobe nbd +MEMORY="1000" +CFG="" +MPTG=4 +MU="M" +EXEC="/usr/share/sdfs/jsvc" +PF="sdfs-ncfg.pid" +if [[ $1:0:1} == '-' ]]; then +while getopts ":v:" opt; do + case $opt in + v) + CFG="/etc/sdfs/$OPTARG-volume-cfg.xml" + PF="$OPTARG.pid" + ;; + z) + MEM=$OPTARG + ;; + :) + echo "Option -$OPTARG requires an argument." >&2 + exit 1 + ;; + esac +done +else + CFG="/etc/sdfs/$1-volume-cfg.xml" +fi + +if [ ! -n "$MEM" ]; then +if [ -n "$CFG" ] && [ -f "$CFG" ]; then + ac=$(echo 'cat //subsystem-config/local-chunkstore/@allocation-size' | xmllint --shell "$CFG" | grep -v ">" | cut -f 2 -d "=" | tr -d \"); + MEMORY=$(((ac/10737418240*MPTG)+1000)) +fi +else + MEMORY=$MEM +fi + +LD_PRELOAD="/usr/share/sdfs/bin/libfuse.so.2" $EXEC -server -outfile '&1' -errfile '&2' -Djava.library.path=/usr/share/sdfs/bin/ -home /usr/share/sdfs/bin/jre -Dorg.apache.commons.logging.Log=fuse.logging.FuseLog\ + -Dfuse.logging.level=INFO -Xmx$MEMORY$MU -Xms$MEMORY$MU \ +-XX:+DisableExplicitGC -pidfile /var/run/$PF -XX:+UseG1GC -Djava.awt.headless=true \ + -cp /usr/share/sdfs/lib/commons-daemon-1.0.15.jar:/usr/share/sdfs/lib/sdfs.jar:/usr/share/sdfs/lib/* org.opendedup.buse.sdfsdev.SDFSVolMgr "$@" diff --git a/install-packages/deb/usr/share/sdfs/readme.txt b/install-packages/deb/usr/share/sdfs/readme.txt index d4f2d46a7..e66bdf2eb 100644 --- a/install-packages/deb/usr/share/sdfs/readme.txt +++ b/install-packages/deb/usr/share/sdfs/readme.txt @@ -1,221 +1,231 @@ -What is this? - - A clustered deduplicated file system. - -Requirements - -System Requirements - - 1. x64 Linux Distribution. The application was tested and developed on ubuntu 13.1.0 and CentOS 6.5 - 3. At least 4 GB of RAM - 4. Java 7 - available at https://jdk7.dev.java.net/ - -Optional Packages - - * NFS Kernel Server - * LIO - for ISCSI support - -Installation - -Ubuntu/Debian (Ubuntu 13.10) - - Step 1: Install Java JRE - - sudo apt-get install openjdk-7-jre-headless - - Step 2: Install SDFS File System - - wget http://www.opendedup.org/downloads/sdfs-2.0-RC1_amd64.deb - sudo dpkg -i sdfs-2.0-RC1_amd64.deb - - Step 3: Change the maximum number of open files allowed - - echo "* hardnofile 65535" >> /etc/security/limits.conf - echo "* soft nofile 65535" >> /etc/security/limits.conf - exit - Step 4: Log Out and Proceed to Initialization Instructions - -CentOS/RedHat (Centos 6.5) - - Step 1: Log in as root - - Step 2: Install Java JRE - - yum install java - - Step 3: Install the SDFS File System - - wget http://www.opendedup.org/downloads/SDFS-2.0.0-RC1.x86_64.rpm - rpm -iv SDFS-2.0.0-RC1.x86_64.rpm - - Step 4: Change the maximum number of open files allowed - - echo "* hardnofile 65535" >> /etc/security/limits.conf - echo "* soft nofile 65535" >> /etc/security/limits.conf - exit - - Step 5: Disable the IPTables firewall - - service iptables save - service iptables stop - chkconfig iptables off - - Step 6: Log Out and Proceed to Initialization Instructions - -Initialization Instructions for Standalone Volumes - - Step 1: Log into the linux system as root or use sudo - - Step 2: Create the SDFS Volume. This will create a volume with 256 GB of capacity using a 4K block size. - - sudo mkfs.sdfs --volume-name=pool0 --volume-capacity=256GB - - Step 3: Create a mount point on the filesystem for the volume - - sudo mkdir /media/pool0 - - Step 4: Mount the Volume - - sudo mount.sdfs pool0 /media/pool0/ & - -Initialization Instructions for A Multi-Node Configuration - - In a mult-node cluster there are two components : - - * The Dedup Storage Engine (DSE) - This is the server/service that store all unique blocks. For redundancy, you should have at least two of these. They can live on the same physical server as other components but this is not recommended for redundancy - * The File System Service (FSS) - This is the server/service that mounts the sdfs volume. Multiple FSS services each serving its own volume can live in the same cluster. - Prerequisits : - - * Static IP address and unique name for all nodes: All nodes should have a static IP address to reduce network issues. - * Disable host firewalls : IPTables should be disabled for testing and then enabled again if needed with open ports for multicast and udp. - * Network Communication : Multicast communication is the default configuration. All nodes in the cluster will need to be able to talk to eachother using multicast unless the Jgroups configuration is modified.Multicast communication is usually not a problem on the same subnet but can be an issue between subnets or over wans - * Fast Low Latency Network : SDFS will work within most networks but speed will suffer is network speeds low or network latency is high. 1Gb/s or above dedicated network is recommeded. - * Update your /etc/sysctl.conf to improve network IO performance. Below are some recommeded options : - vm.swappiness=1 - net.core.rmem_max=254800000 - net.core.wmem_max=254800000 - net.core.rmem_default=254800000 - net.core.wmem_default=254800000 - net.core.optmem_max=25480000 - net.ipv4.tcp_timestamps=0 - net.ipv4.tcp_sack=0 - net.core.netdev_max_backlog=250000 - net.ipv4.tcp_mem=25480000 25480000 25480000 - net.ipv4.tcp_rmem=4096 87380 25480000 - net.ipv4.tcp_wmem=4096 65536 25480000 - net.ipv4.tcp_low_latency=1 - Creating and Starting a DSE - -The following steps will create 2 DSEs on server1 and server2 - - Step 1: Create a DSE on Server1 using a 4K block size and 200GB of capacity and cluster node id of "1" - - mkdse --dse-name=sdfs --dse-capacity=200GB --cluster-node-id=1 - - Step 2: Edit the /etc/sdfs/jgroups.cfg.xml and add the bind_addr attribute with Server1's IP address to the tag. - - tag. - - /sys/devices/virtual/net/$INTERFACE/bridge/multicast_querier - fi - - Testing multicast support on nodes in the cluster - - The jgroups protocol includes a nice tool to verify multicast is working on all nodes. Its an echo tool and sends messages from a sender to a reciever - - On the receiver run - - java -cp /usr/share/sdfs/lib/jgroups-3.4.1.Final.jar org.jgroups.tests.McastReceiverTest -mcast_addr 231.12.21.132 -port 45566 - - On the sender run - - java -cp /usr/share/sdfs/lib/jgroups-3.4.1.Final.jar org.jgroups.tests.McastSenderTest -mcast_addr 231.12.21.132 -port 45566 - - Once you have both sides running type a message on the sender and you should see it on the receiver after you press enter. You may also want to switch rolls to make sure multicast works both directions. - - take a look at http://docs.jboss.org/jbossas/docs/Clustering_Guide/4/html/ch07s07s11.html for more detail. - - Further reading: - - Take a look at the administration guide for more detail. http://www.opendedup.org/sdfs-20-administration-guide - - Ask for Help - - If you still need help check out the message board here https://groups.google.com/forum/#!forum/dedupfilesystem-sdfs-user-discuss +What is this? + + A clustered deduplicated file system. + +Requirements + +System Requirements + + 1. x64 Linux Distribution. The application was tested and developed on ubuntu 14.0.4 and CentOS 7 + 2. At least 4 GB of RAM + 3. Java 8 - available at https://jdk8.dev.java.net/ + 4. JSVC (http://commons.apache.org/proper/commons-daemon/jsvc.html) + +Optional Packages + + * NFS Kernel Server + * LIO - for ISCSI support + +Installation + +Ubuntu/Debian (Ubuntu 14.04+) + + Step 1: Download the latest sdfs version + wget http://opendedup.org/downloads/sdfs-latest.deb + + Step 2: Install sdfs and dependencies + sudo apt-get install fuse libfuse2 ssh openssh-server jsvc libxml2-utils + sudo dpkg -i sdfs-latest.deb + + Step 3: Change the maximum number of open files allowed + echo "* hardnofile 65535" >> /etc/security/limits.conf + echo "* soft nofile 65535" >> /etc/security/limits.conf + exit + Step 5: Log Out and Proceed to Initialization Instructions + +CentOS/RedHat (Centos 7.0+) + + Step 1: Download the latest sdfs version + wget http://opendedup.org/downloads/sdfs-latest.rpm + + Step 2: Install sdfs and dependencies + yum install jsvc libxml2 java-1.8.0-openjdk + rpm -iv --force sdfs-latest.rpm + + Step 3: Change the maximum number of open files allowed + echo "* hardnofile 65535" >> /etc/security/limits.conf + echo "* soft nofile 65535" >> /etc/security/limits.conf + exit + Step 5: Log Out and Proceed to Initialization Instructions + + Step 6: Disable the IPTables firewall + + service iptables save + service iptables stop + chkconfig iptables off + + Step 7: Log Out and Proceed to Initialization Instructions + +Initialization Instructions for Standalone Volumes + + + Step 1: Log into the linux system as root or use sudo + + Step 2: Create the SDFS Volume. This will create a volume with 256 GB of capacity using a Variable block size. + **Local Storage** + sudo mkfs.sdfs --volume-name=pool0 --volume-capacity=256GB + + **AWS Storage** + sudo mkfs.sdfs --volume-name=pool0 --volume-capacity=1TB --aws-enabled true --cloud-access-key --cloud-secret-key --cloud-bucket-name + + **Azure Storage** + sudo mkfs.sdfs --volume-name=pool0 --volume-capacity=1TB --azure-enabled true --cloud-access-key --cloud-secret-key --cloud-bucket-name + + **Google Storage** + sudo mkfs.sdfs --volume-name=pool0 --volume-capacity=1TB --google-enabled true --cloud-access-key --cloud-secret-key --cloud-bucket-name + + + Step 3: Create a mount point on the filesystem for the volume + + sudo mkdir /media/pool0 + + Step 4: Mount the Volume + + sudo mount -t sdfs pool0 /media/pool0/ + + Set 5: Add the filesystem to fstab + pool0 /media/pool0 sdfs defaults 0 0 + + + +Initialization Instructions for A Multi-Node Configuration + + In a mult-node cluster there are two components : + + * The Dedup Storage Engine (DSE) - This is the server/service that store all unique blocks. For redundancy, you should have at least two of these. They can live on the same physical server as other components but this is not recommended for redundancy + * The File System Service (FSS) - This is the server/service that mounts the sdfs volume. Multiple FSS services each serving its own volume can live in the same cluster. + Prerequisits : + + * Static IP address and unique name for all nodes: All nodes should have a static IP address to reduce network issues. + * Disable host firewalls : IPTables should be disabled for testing and then enabled again if needed with open ports for multicast and udp. + * Network Communication : Multicast communication is the default configuration. All nodes in the cluster will need to be able to talk to eachother using multicast unless the Jgroups configuration is modified.Multicast communication is usually not a problem on the same subnet but can be an issue between subnets or over wans + * Fast Low Latency Network : SDFS will work within most networks but speed will suffer is network speeds low or network latency is high. 1Gb/s or above dedicated network is recommeded. + * Update your /etc/sysctl.conf to improve network IO performance. Below are some recommeded options : + vm.swappiness=1 + net.core.rmem_max=254800000 + net.core.wmem_max=254800000 + net.core.rmem_default=254800000 + net.core.wmem_default=254800000 + net.core.optmem_max=25480000 + net.ipv4.tcp_timestamps=0 + net.ipv4.tcp_sack=0 + net.core.netdev_max_backlog=250000 + net.ipv4.tcp_mem=25480000 25480000 25480000 + net.ipv4.tcp_rmem=4096 87380 25480000 + net.ipv4.tcp_wmem=4096 65536 25480000 + net.ipv4.tcp_low_latency=1 + Creating and Starting a DSE + +The following steps will create 2 DSEs on server1 and server2 + + Step 1: Create a DSE on Server1 using a 4K block size and 200GB of capacity and cluster node id of "1" + + mkdse --dse-name=sdfs --dse-capacity=200GB --cluster-node-id=1 + + Step 2: Edit the /etc/sdfs/jgroups.cfg.xml and add the bind_addr attribute with Server1's IP address to the tag. + + tag. + + /sys/devices/virtual/net/$INTERFACE/bridge/multicast_querier + fi + + Testing multicast support on nodes in the cluster + + The jgroups protocol includes a nice tool to verify multicast is working on all nodes. Its an echo tool and sends messages from a sender to a reciever + + On the receiver run + + java -cp /usr/share/sdfs/lib/jgroups-3.4.1.Final.jar org.jgroups.tests.McastReceiverTest -mcast_addr 231.12.21.132 -port 45566 + + On the sender run + + java -cp /usr/share/sdfs/lib/jgroups-3.4.1.Final.jar org.jgroups.tests.McastSenderTest -mcast_addr 231.12.21.132 -port 45566 + + Once you have both sides running type a message on the sender and you should see it on the receiver after you press enter. You may also want to switch rolls to make sure multicast works both directions. + + take a look at http://docs.jboss.org/jbossas/docs/Clustering_Guide/4/html/ch07s07s11.html for more detail. + + Further reading: + + Take a look at the administration guide for more detail. http://www.opendedup.org/sdfs-20-administration-guide + + Ask for Help + + If you still need help check out the message board here https://groups.google.com/forum/#!forum/dedupfilesystem-sdfs-user-discuss \ No newline at end of file diff --git a/install-packages/deb/usr/share/sdfs/sdfscli b/install-packages/deb/usr/share/sdfs/sdfscli index b07c237d5..71c5174cb 100755 --- a/install-packages/deb/usr/share/sdfs/sdfscli +++ b/install-packages/deb/usr/share/sdfs/sdfscli @@ -1 +1 @@ -java -classpath /usr/share/sdfs/lib/sdfs.jar:/usr/share/sdfs/lib/* org.opendedup.sdfs.mgmt.cli.SDFSCmdline "$@" +/usr/share/sdfs/bin/jre/bin/java -classpath /usr/share/sdfs/lib/sdfs.jar:/usr/share/sdfs/lib/* org.opendedup.sdfs.mgmt.cli.SDFSCmdline "$@" diff --git a/install-packages/deb/usr/share/sdfs/sdfsreplicate b/install-packages/deb/usr/share/sdfs/sdfsreplicate index 07b7caad5..44d953d87 100755 --- a/install-packages/deb/usr/share/sdfs/sdfsreplicate +++ b/install-packages/deb/usr/share/sdfs/sdfsreplicate @@ -1 +1 @@ -java -classpath /usr/share/sdfs/lib/sdfs.jar:/usr/share/sdfs/lib/* org.opendedup.sdfs.replication.ReplicationService "$@" +/usr/share/sdfs/bin/jre/bin/java -classpath /usr/share/sdfs/lib/sdfs.jar:/usr/share/sdfs/lib/* org.opendedup.sdfs.replication.ReplicationService "$@" diff --git a/install-packages/windows/readme-win32.txt b/install-packages/windows/readme-win32.txt index 79259435e..2441f3508 100644 --- a/install-packages/windows/readme-win32.txt +++ b/install-packages/windows/readme-win32.txt @@ -15,7 +15,7 @@ Getting Started Step 1: Install the . It can be downloaded from: http://www.opendedup.org/downloads/sdfs-latest-setup.exe - Step 1: Create an sdfs file system. + Step 2: Create an sdfs file system. To create and SDFS file System you must run the mksdfs command from within the SDFS binary directory. Make sure you have "Full Control permissions to the "c:\program files\sdfs" directory. @@ -25,17 +25,17 @@ Getting Started e.g. mksdfs --volume-name=sdfs_vol1 --volume-capacity=100GB - Step 2: Mount the sdfs + Step 3: Mount the sdfs To mount SDFS run the following command: mountsdfs -v -m e.g. mountsdfs -v sdfs_vol1 -m S - Step 3: Unmount volume + Step 4: Unmount volume To unmount the volume run sdfscli --shutdown - Step 3: Mount sdfs as a service + Step 5: Mount sdfs as a service To have sdfs run on boot you will need to use the sc tool to add each volume you want to mount as a service. The service will not show that it is running because mountsdfs is not yet built as a service, but the volume will be mounted. Below is how you add an sdfs volume as a service. diff --git a/src/readme-win32.txt b/src/readme-win32.txt index fe12ed13e..128d0e54c 100644 --- a/src/readme-win32.txt +++ b/src/readme-win32.txt @@ -4,18 +4,18 @@ What is this? System Requirements - * Windows (7,XP,Vista,2003,2008) Distribution. The application was tested and developed on Windows 7 (32bit) - * Dokan 5.3+. Dokan can be downloaded from http://dokan-dev.net/en/ + * Windows x64 (7,10,2008,2012) Distribution. The application was tested and developed on Windows 10 (64bit) + * Dokany 8.0+ (Included in package) * 2 GB of Available RAM - * Java 7 (32 bit) - included in package. + * Java 8 (64 bit) - included in package. Getting Started - Step 1: Install the Dokan library and driver. It can be downloaded from: - http://dokan-dev.net/wp-content/uploads/dokaninstall_053.exe + Step 1: Install the . It can be downloaded from: + http://www.opendedup.org/downloads/sdfs-latest-setup.exe - Step 1: Create an sdfs file system. + Step 2: Create an sdfs file system. To create and SDFS file System you must run the mksdfs command from within the SDFS binary directory. Make sure you have "Full Control permissions to the "c:\program files\sdfs" directory. @@ -25,123 +25,36 @@ Getting Started e.g. mksdfs --volume-name=sdfs_vol1 --volume-capacity=100GB - Step 2: Mount the sdfs + Step 3: Mount the sdfs To mount SDFS run the following command: mountsdfs -v -m e.g. mountsdfs -v sdfs_vol1 -m S -Known Limitation(s) + Step 4: Unmount volume + To unmount the volume run sdfscli --shutdown - Testing has been limited thus far. Please test and report bugs - Graceful exit if physical disk capacity is reached. ETA : Will be implemented shortly - Maximum individual filesize within sdfs currently 250GB at 4k chunks and multiples of that at higher chunk sizes. - Only one SDFS volume can be mounted at a time. + Step 5: Mount sdfs as a service + To have sdfs run on boot you will need to use the sc tool to add each volume you want to mount as a service. The service will not show + that it is running because mountsdfs is not yet built as a service, but the volume will be mounted. Below is how you add an sdfs volume as a + service. + sc create sdfs binPath="\"C:\Program Files\sdfs\mountsdfs.exe\" -v sdfs -m -cp" DisplayName="SDFS Drive" start=auto + e.g. + sc create sdfss binPath="\"C:\Program Files\sdfs\mountsdfs.exe\" -v sdfs_vol1 -m s -cp" DisplayName="SDFS s Drive" start=auto -Performance : +Using Cloud storage with SDFS - By default SDFS on Windows is single threaded and this will be changed in the future once more testing is done. Based on testing, a single - threaded SDFS volume will write at about 15 MB/s. - To enable faster throughput multithreading can be enabled either when the volume is created on within the XML file. To enable - multithreading at volume creation add "--io-safe-close=false" as a command line option to mksdfs. Otherwise edit the xml configuration - file (-volume-cfg.xml) within c:\program files\sdfs\etc and set "io-saf-close=false" . - - Example to create a SDFS volume with multi-threaded support: - - mksdfs --volume-name= --volume-capacity= --io-safe-close=false - - Safe close prevents the file from closing when commanded by the filesystem, but is instead closed based on inactivity. Setting this to - "false" could cause corruption if the file is still open while the system crashes. +SDFS can use cloud object storage to store data. To enable this make the volume with the following commands: -Data Removal - - SDFS uses a batch process to remove unused blocks of hashed data.This process is used because the file-system is - decoupled from the back end storage (ChunkStore) where the actual data is held. As hashed data becomes stale they - are removed from the chunk store. The process for determining and removing stale chunks is as follows. + **AWS Storage** + mksdfs --volume-name=pool0 --volume-capacity=1TB --aws-enabled true --cloud-access-key --cloud-secret-key --cloud-bucket-name - 1. SDFS file-system informs the ChunkStore what chunks are currently in use. This happens when - chunks are first created and then every 6 hours on the hour after that. - 2. The Chunk Store checks for data that has not been claimed, in the last two days, by the SDFS file system - every 24 hours. - 3. The chunks that have not been claimed in the last 25 hours are put into a pool and overwritten as new data - is written to the ChunkStore. - - All of this is configurable and can be changed after a volume is written to. Take a look at cron format for more details. - + **Azure Storage** + mksdfs --volume-name=pool0 --volume-capacity=1TB --azure-enabled true --cloud-access-key --cloud-secret-key --cloud-bucket-name + + **Google Storage** + mksdfs --volume-name=pool0 --volume-capacity=1TB --google-enabled true --cloud-access-key --cloud-secret-key --cloud-bucket-name -Tips and Tricks - - There are plenty of options to create a file system and to see them run "./mkfs.sdfs --help". - - Chunk Size: - - The most notable option is --io-chunk-size. The option --io-chunk-size sets the size of chunks that are hashed and can only - be changed before the file system is mounted to for the first time. The default setting is 128k but can be set as low as 4k. - The size of chucks determine the efficient at which files will be deduplicated at the cost of RAM. As an example a 4k chunk - size is perfect for Virtual Machines (VMDKs) because it matches the cluster size of most guest os file systems but can cost - as much as 8GB of RAM per 1TB to store. In contrast setting the chunk size to 128k is perfect of archived data, such as backups, - and will allow you to store as much as 32TB of data with the same 8GB of memory. - - To create a volume that will store VMs (VMDK files) create a volume using 4k chunk size as follows: - mksdfs --volume-name=sdfs_vol1 --volume-capacity=150GB --io-chunk-size=4k --io-max-file-write-buffers=150 - - - File Placement: - - Deduplication is IO Intensive. SDFS, by default writes data to c:\program files\sdfs. MAKE SURE YOU HAVE READ AND WRITE PERMISSIONS - TO THIS DIRECTORY SDFS does a lot of writes went persisting data and a lot of random io when reading data. For high IO intensive - applications it is suggested that you split at least the chunk-store-data-location and chunk-store-hashdb-location onto fast and - separate physical disks. From experience these are the most io intensive stores and could take advantage of faster IO. - - Inline vs Batch based deduplication: - - SDFS provides the option to do either inline or batch based deduplication. Inline deduplication stores the data to the deduplicated - chunkstore in realtime. Batch based deduplication stores data to disk as a normal file unless a match is found in the chunk store to - a previously persisted deduplicated chunk of data. Inline deduplication is perfect for backup but not as good for live data such as - VMs or databases because there is a high rate of change for those types of files. Batch based deduplication has a lot of overhead since - there is eventually twice as much IO per file, one to store it to disk and once to dedup it. By default, inline is enabled, but can be - disabled with the option "--io-dedup-files=false" when creating the volume. - - When this option is set to false all data will be stored in its origional format until deduplication is set on a per file basis. To set - deduplication to on, run the command "sdfscli --dedup-file=true --file-path=". Conversely, setting - "sdfscli --dedup-file=true --file-path=" will disable deduplication of a specific file from that point forward. - - - - Snapshots: - - SDFS provides snapshot functions for files and folders. The snapshot command is "sdfscli --snapshot --file-path= --snapshot-path=". The destination path is relative to the mount point of the sdfs filesystem. - - Other options and extended attributes: - - SDFS keeps deduplication rates and IO per file. To get IO statistics run "sdfscli --file-info --file-path=" within the mount point of the sdfs file system. - - Memory: - - The mountsdfs shell script currently allocates up to 2GB of RAM for the SDFS file system. This is fine for SDFS file systems of around - 200GB for 4k chunk size and around 6TB for 128k chunk size. To expand the memory edit the "-Xmx2g" within mount.sdfs script to something - better for your environment. Each stored chunk takes up approximately 33 bytes of RAM. To calculate how much RAM you will need for a - specific volume divide the volume size (in bytes) by the chunk size (in bytes) and multiply that times 33. - - Memory Requirements Calculation: (volume size/chunk size)*33 - - Cloud Storage and Amazon S3 Web Serivce - - It is now possible to store dedup chunks to the Amazon S3 cloud storage service. This will allow you to store unlimited amounts of data - without the need for local storage. AES 256 bit encryption and compression (default) is provided for storing data to AWS. It is suggested - that the chunk size be set to the default (128k) to allow for maximum compression and fewest round trips for data. - To Setup AWS enable dedup volume follow these steps: - 1. Go to http://aws.amazon.com and create an account. - 2. Sign up for S3 data storage - 3. Get your Access Key ID and Secret Key ID. - 4. Make an SDFS volume using the following parameters: - mksdfs --volume-name= --volume-capacity= --aws-enabled=true --aws-access-key= --aws-bucket-name= --aws-secret-key= --chunk-store-encrypt=true - 5. Mount volume and go to town! - mountsdfs -v -m - - -9/12/10 \ No newline at end of file +For more information take a look at the opendedup website at www.opendedup.org \ No newline at end of file diff --git a/src/readme.txt b/src/readme.txt index e66bdf2eb..10eaf99c1 100644 --- a/src/readme.txt +++ b/src/readme.txt @@ -8,8 +8,6 @@ System Requirements 1. x64 Linux Distribution. The application was tested and developed on ubuntu 14.0.4 and CentOS 7 2. At least 4 GB of RAM - 3. Java 8 - available at https://jdk8.dev.java.net/ - 4. JSVC (http://commons.apache.org/proper/commons-daemon/jsvc.html) Optional Packages @@ -31,7 +29,7 @@ Ubuntu/Debian (Ubuntu 14.04+) echo "* hardnofile 65535" >> /etc/security/limits.conf echo "* soft nofile 65535" >> /etc/security/limits.conf exit - Step 5: Log Out and Proceed to Initialization Instructions + Step 4: Log Out and Proceed to Initialization Instructions CentOS/RedHat (Centos 7.0+) @@ -46,15 +44,15 @@ CentOS/RedHat (Centos 7.0+) echo "* hardnofile 65535" >> /etc/security/limits.conf echo "* soft nofile 65535" >> /etc/security/limits.conf exit - Step 5: Log Out and Proceed to Initialization Instructions + Step 4: Log Out and Proceed to Initialization Instructions - Step 6: Disable the IPTables firewall + Step 5: Disable the IPTables firewall service iptables save service iptables stop chkconfig iptables off - Step 7: Log Out and Proceed to Initialization Instructions + Step 6: Log Out and Proceed to Initialization Instructions Initialization Instructions for Standalone Volumes @@ -228,4 +226,4 @@ Troubleshooting and other Notes Ask for Help - If you still need help check out the message board here https://groups.google.com/forum/#!forum/dedupfilesystem-sdfs-user-discuss \ No newline at end of file + If you still need help check out the message board here https://groups.google.com/forum/#!forum/dedupfilesystem-sdfs-user-discuss