Skip to content

Commit

Permalink
Fix issue with logger library and windows. Tidy README.md
Browse files Browse the repository at this point in the history
  • Loading branch information
Seth Van Buren authored and Seth Van Buren committed May 23, 2019
1 parent 48b39f7 commit 9cfc6d6
Show file tree
Hide file tree
Showing 4 changed files with 29 additions and 5 deletions.
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,5 @@ s3kor.linux
s3kor.mac
s3kor.linux.gz
s3kor
s3kor.windows
s3kor.windows.gz
8 changes: 6 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,15 @@ linux:
env GOOS=linux GOARCH=amd64 go build -o s3kor.linux
gzip -fk9 s3kor.linux

windows:
env GOOS=windows GOARCH=amd64 go build -o s3kor.windows
gzip -fk9 s3kor.windows

clean:
rm s3kor.linux s3kor.linux.gz
rm s3kor.linux s3kor.linux.gz s3kor.windows s3kor.windows.gz s3kor

publish-test:
goreleaser --snapshot --rm-dist

publish:
goreleaser --rm-dist --skip-validate
goreleaser --rm-dist --skip-validate
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ Args:
<destination> file or s3 location
```

Tha maximum. concurrent uploads (`--concurrent` or `-c`) is dependent not only on your upload bandwidth but also the maximum open file limits per process on your system and the performance of the soucre drive.
Tha maximum concurrent uploads (`--concurrent` or `-c`) is dependent not only on your upload bandwidth but also the maximum open file limits per process on your system and the performance of the soucre drive.

You can check your file limits in linux, macos and other flavour of OS with `ulimit -n`. Changing this limit in the os is possible and not always dangerous. Instructions on how to change it vary between OS so they are not described here. `s3kor` impacts these limits both in walking the file system and uploading the file so there is not a 1 to 1 correlation between the max limit ond the value you pass to `--concurrent`. Try to pass `s3kor` a max value that is about 20% less than the systems max limit value.

Expand Down
22 changes: 20 additions & 2 deletions s3kor.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,9 @@ package main
import (
"fmt"
"io/ioutil"
"net/url"
"os"
"runtime"

"github.com/aws/aws-sdk-go/service/s3/s3manager"

Expand Down Expand Up @@ -63,6 +65,12 @@ var (
date = "unknown"
)

///Needed to workaround abug with zap logger and daft windows file paths/names
func newWinFileSink(u *url.URL) (zap.Sink, error) {
// Remove leading slash left by url.Parse()
return os.OpenFile(u.Path[1:], os.O_WRONLY|os.O_APPEND|os.O_CREATE, 0644)
}

func main() {
//Lets keep a track on how long things are taking us
//startTime := time.Now()
Expand All @@ -86,9 +94,19 @@ func main() {

}

config.OutputPaths = []string{
logFile.Name(),
//workaround for windows file paths and names

if runtime.GOOS == "windows" {
zap.RegisterSink("winfile", newWinFileSink)
config.OutputPaths = []string{
"winfile:///" + logFile.Name(),
}
} else {
config.OutputPaths = []string{
logFile.Name(),
}
}

logger, _ := config.Build()
zap.ReplaceGlobals(logger)
zap.RedirectStdLog(logger)
Expand Down

0 comments on commit 9cfc6d6

Please sign in to comment.