Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Missing routes for Remote-Remote #100

Open
mrocklin opened this issue Feb 16, 2015 · 1 comment
Open

Missing routes for Remote-Remote #100

mrocklin opened this issue Feb 16, 2015 · 1 comment
Milestone

Comments

@mrocklin
Copy link
Member

We lack direct routes to copy data between remote computers without touching the local system.

into('ssh://A:myfile.csv', 'ssh://A:myfile2.csv')  # Should be a remote local copy
into('ssh://A:myfile.csv', 'ssh://B:myfile.csv')  # Should result in a remote call to scp
into('hdfs://A:myfile.csv', 'ssh://A.myfile.csv')  # Should result in hadoop fs calls
into('ssh://A:myfile.csv', 'hdfs://A.myfile.csv')  # Should result in hadoop fs calls
...

When interacting with AWS the inter-node bandwidth is very high. It'd be nice to leverage this instead of pulling down and pushing up from local.

@mrocklin
Copy link
Member Author

One tricky thing is that we'll probably have to pass credentials up securely.

@cpcloud cpcloud modified the milestones: 0.3.1, 0.3.2 Mar 6, 2015
@cpcloud cpcloud modified the milestone: 0.3.2 Mar 19, 2015
@cpcloud cpcloud modified the milestone: 0.3.4 Jun 22, 2015
@cpcloud cpcloud modified the milestones: 0.3.4, 0.4.0 Sep 15, 2015
@cpcloud cpcloud modified the milestones: 0.4.0, 0.3.5 Oct 5, 2015
@cpcloud cpcloud modified the milestones: 0.4.0, 0.4.1 Dec 4, 2015
@kwmsmith kwmsmith modified the milestones: 0.4.1, 0.4.2, 0.5.0 Feb 2, 2016
@kwmsmith kwmsmith removed this from the 0.4.2 milestone Feb 5, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants