0.3.2

What's Changed

• IBM Support by @sarahwooders in #731
• Resolve redundant GCP logs in load_gcp_config by @mech-a in #786
• [Fix] Azure init issues by @zizhong in #787
• Delete test buckets during integration testing by @parasj in #798
• Programmable Gateways by @sarahwooders in #804
• [Fix] exit before transfer all the files. by @killerdbob in #813
• Support private versus public IPs for gateways by @sarahwooders in #817
• Add integration tests with multiple file sizes. by @xutingl in #816
• Fix Azure Quota API call error message by @simon-mo in #821
• Fix integration tests by disabling IBM for now by @sarahwooders in #822
• Fix single region transfers (same source/destinations) by @sarahwooders in #826
• Implement azure private ip function by @sarahwooders in #828
• Update google cloud auth action to v1 by @sarahwooders in #830
• Add Nix flake for creating development environments by @shadaj in #833
• Updated integration tests and local transfers fix by @sarahwooders in #831
• Don't search host directories for private keys when connecting to a cloud instance by @shadaj in #837
• Fix HTTP connection error for long running transfers by @sarahwooders in #842
• added null check to azure vnet cleanup by @abiswal2001 in #851
• fall back to smaller vm if necessary by @xTRam1 in #815
• Cloudflare R2 Integration by @sarahwooders in #856
• 0.3.1 release by @sarahwooders in #882
• Remove dependencies on specific cloud provider packages in TransferJob and update test deprovision bucket names by @sarahwooders in #884
• Update poetry-publish-nightly.yml by @abiswal2001 in #886
• reverting nightly to fix docker image mismatch by @abiswal2001 in #887
• Add back compression and encryption by @lynnliu030 in #877
• Add integration tests, re-enable azure multipart, and include firewall rule deletion for GCP deprovision by @sarahwooders in #891
• Run multipart for Azure integration tests by @sarahwooders in #892
• Remove cloudflare config print by @sarahwooders in #893

New Contributors

• @mech-a made their first contribution in #786
• @killerdbob made their first contribution in #813
• @shadaj made their first contribution in #833

Full Changelog: 0.3.1...0.3.2

skyplane-0.3.1

This PR fixes a bug in 0.3.0 where transfers would break due to an error in the Skyplane Gateway.

skyplane-0.3.0

What's Changed

• Clean AWS provisioning logic to prepare for debugging concurrent Skyplane usage by @parasj in #610
• Add flag before deprovisioning networks by @parasj in #615
• Clean up leaked resources upon VM provisioning errors for AWS, GCP and Azure by @parasj in #617
• Prefix AWS keys with UUID, tag VMs with UUID by @parasj in #616
• Organize imports, run autoflake and clean up logic around gateway cleanup by @parasj in #619
• Linked to paper by @vdm in #620
• Simplify README by @sarahwooders in #622
• Fix #596 and #598: error profiling throughput grid on aws and gcp by @lynnliu030 in #605
• Run jobs on pull_request_target to support Github Actions for external contributors by @parasj in #633
• Use questionary to allow selection of multiple Azure subscriptions by @parasj in #632
• Run pytest when label is added or for contributors by @parasj in #634
• azure: add 'tags' argument to provision_instance by @phi-line in #630
• Add support for Python 3.11 by @parasj in #639
• Add skyplane azure check command by @parasj in #647
• Fix broken Azure path logic by @parasj in #650
• [draft] Remove setup.py with editable support by @parasj in #640
• Cleanup __init__.py by @parasj in #658
• [API] Meta API tracking PR by @parasj in #652
• Move cloud CLI tools to common file by @parasj in #660
• Interface for on-prem and pre-signed urls by @ShishirPatil in #659
• Update exception chaining logic to remove long traceback by @parasj in #661
• Copy mime types with transfers by @parasj in #655
• Move compute requirements under single package by @parasj in #665
• Move GCP networking + pricing logic out of cloud provider by @parasj in #614
• Use short UUIDs to limit length of provisioned resources by @parasj in #664
• Migrate chunk IDs from ints to UUIDs by @parasj in #656
• [bug] Fix integration test by @parasj in #670
• [API] Performance Logging by @JasonDing0401 in #669
• [API] fix client id issue by @JasonDing0401 in #671
• [API] ML training example by @JasonDing0401 in #676
• Fix Python 3.7 for config_paths by @parasj in #668
• Nit: update gateway_daemon.py for inflight chunks by @ShishirPatil in #682
• Clean up API definitions by @parasj in #686
• Programmable gateways implemented with queue-only inter-process communication by @sarahwooders in #678
• Small reformatting changes from automatic scripts by @sarahwooders in #688
• Native cmd fallback by @ShishirPatil in #694
• [broadcast] Fix issues with gateway merge to main by @parasj in #689
• Use PyPI release of Skyplane for CI to deprovision VMs by @parasj in #696
• [ci] Fix deprovision bug on main by @parasj in #699
• [api] Migrate CLI over the API by @abiswal2001 in #677
• Add user configurable option for default location for multi-region buckets by @parasj in #711
• disable OS Login by default for GCP VMs by @troycarlson in #708
• fix: mismatched console markup by @troycarlson in #720
• Handle lost instance profiles during deprovisioning by @parasj in #724
• Add quotes around pip install command for alternative shells like Zsh by @parasj in #730
• [hotfix] Fix integration tests on main by @parasj in #713
• [API] Documentations by @JasonDing0401 in #712
• Restore ILP support branch by @parasj in #739
• [bug] Azure auth config client id update by @abiswal2001 in #742
• Prompt users to login, while setting up azure by @ShishirPatil in #744
• [Fix] the azure auth issue by @zizhong in #743
• [Fix] Client in CLI uses correct transfer config by @zizhong in #747
• [Fix] gateway requirements by @zizhong in #749
• [fix] minor get instance bug in azure by @abiswal2001 in #752
• [fix] add back copy_log by @zizhong in #750
• [Fix] gateway auth config and azure download with offset by @zizhong in #751
• [fix] Adjusting reinit by @abiswal2001 in #753
• [fix] Copy gateway logs on interrupt by @abiswal2001 in #754
• [#755] Add support for SEV provisioning in Azure by @zizhong in #756
• GCP multipart completion fix + multi-project service credentials by @sarahwooders in #757
• Updated keyboard interrupt warning by @abiswal2001 in #768
• Reformat code and pass pytype by @parasj in #770
• Clean up confusing variable in dataplane.py by @parasj in #769
• Prototype Airflow operator by @parasj in #740
• Update documentation with API information and Tutorials by @sarahwooders in #771
• Preliminary support for on-prem transfers with HDFS and POSIX path support by @ShishirPatil in #735
• Add object store interface to API client + tests by @sarahwooders in #773
• Add cost estimation for dataplane queued jobs by @sarahwooders in #776
• Download gateway logs with --debug flag by @sarahwooders in #778
• Persist quota information on disk during skyplane init by @simon-mo in #772
• Bump poetry version to 1.4.0 for releases by @parasj in #779

New Contributors

• @vdm made their first contribution in #620
• @phi-line made their first contribution in #630
• @troycarlson made their first contribution in #708
• @zizhong made their first contribution in #743
• @simon-mo made their first contribution in #772

Full Changelog: 0.2.1...0.3.0

skyplane-0.2.1

What's Changed

• Mitigate Azure object store stragglers by limiting gateway connections by @parasj in #594
• Update documentation with link to benchmarks and architecture by @parasj in #601
• Improve dependency management for Skyplane by @parasj in #602
• Fix replicate random by printing statistics by @parasj in #593
• #585: migrate to private ips by @lynnliu030 in #607
• Fix Azure core import issue by @parasj in #613

New Contributors

• @lynnliu030 made their first contribution in #607

Full Changelog: 0.2.0...0.2.1

skyplane-0.2.0

What's Changed

Performance improvements

• Multi instance parallelism by @ShishirPatil in #565
• Accelerate multipart transfer requests by @parasj in #578

Bug fixes

• Fixes #560 - aws s3 sync no-follow-symlink by @ShishirPatil in #562
• Fix integration tests w/ bad method rename by @parasj in #574
• Catch all errors during transfer, not just KeyboardInterrupt by @parasj in #577
• Extract cleanup from #578 by @parasj in #583

CLI

• Check for az CLI, update usage message by @parasj in #567
• Partition dependencies by cloud provider by @parasj in #573
• Increase python version for integration test to 3.10 by @parasj in #580

Testing and CI

• [CI] Test multiple versions of Python by @parasj in #576

Full Changelog: 0.1.3...0.2.0

skyplane-0.1.3

Documentation

• Add logo to README by @parasj in #488
• Compress logo to improve page load times by @parasj in #489
• Add OpenGraph metadata to documentation website by @parasj in #494
• Add benchmark documentation to website by @JasonDing0401 in #496
• Update README.md by @HaileyJang in #501
• Update faq.md with requester pays info by @HaileyJang in #538
• Chunking size benchmark to docs by @antonzabreyko in #539
• Clean up code and update contributor docs by @parasj in #556

CI testing

• [CI] Run integration test nightly by @parasj in #526
• [CI] Clean up interfaces with improved object store interface testing by @parasj in #525
• [CI] Unit tests for Azure by @parasj in #529
• [CI] Integration tests for Azure by @parasj in #532
• Fix flaky S3 test by setting default boto3 region by @parasj in #534
• Add delays to object store tests to mitigate flaky AWS test by @parasj in #557

Features

• Use spot instances to reduce transfer costs by @parasj in #513
• [UX] Print better error message when transfer validation fails by @liuxt129 in #535
• Support for Requester Pays by @HaileyJang in #520
• Dispatch multipart transfer requests in parallel by @parasj in #515
• On-prem support for Skyplane by @ShishirPatil in #540
• Small transfer support by @ShishirPatil in #544
• Fix integration test on main, extract fallback CLI commands, add Azure fallback support and add sync fallback support by @parasj in #555

Bug fixes

• Fix edge cases for whole bucket transfers by @parasj in #487
• Fix Docker build issues by @parasj in #498
• Change multipart_min_threshold_mb from 64 to 128 by @JasonDing0401 in #500
• Removing dollar sign from code blocks to mitigate $ copying by @abiswal2001 in #506
• Azure Authentication 2.0 (User Managed Identities) by @abiswal2001 in #505
• reenabling azure verification by @abiswal2001 in #507
• Allow for zero-byte objects by @parasj in #519
• Check if Skyplane config is correct by @parasj in #518
• fixing on-demand bug by @antonzabreyko in #533
• Misc fixes for experiment update by @parasj in #516
• Move enrich_dest_objs to separate func by @abiswal2001 in #527
• Fix: cloud to cloud cli error by @ShishirPatil in #542
• fix:minor bugs by @ShishirPatil in #547
• Dev/shishir/fix by @ShishirPatil in #549
• Remove unused Azure config items from SkyplaneConfig by @abiswal2001 in #522

New Contributors

• @HaileyJang made their first contribution in #501
• @liuxt129 made their first contribution in #535

Full Changelog: 0.1.2...0.1.3

skyplane-0.1.2

What's Changed

• Create codeql-analysis.yml by @parasj in #463
• Support GCP copies to/from unowned buckets by @parasj in #464
• fix pip install awscli by @JasonDing0401 in #466
• fix azure vm resource not found error by @JasonDing0401 in #467
• Fix Sphinx builds by @parasj in #470
• Add Slack link to docs by @sarahwooders in #465
• Set up ReadTheDocs by @parasj in #469
• Fix division by zero error logger message by @JasonDing0401 in #471
• Print a better error upon Azure permission errors by @parasj in #472
• Faster transfer verification via prefixes by @parasj in #475
• Multipart transfer by @JasonDing0401 in #477
• Push docker images to user's fork by @parasj in #473
• Dynamically determine multipart chunk strategy by @parasj in #479
• Clean up object store interfaces by @parasj in #478
• Configure Azure credentials using a service principal instead of DefaultAzureCredential by @parasj in #474

New Contributors

• @JasonDing0401 made their first contribution in #466

Full Changelog: 0.1.1...0.1.2

skyplane-0.1.1

What's Changed

• AWS server provisioning by @parasj in #1
• GCP server provisioner by @parasj in #2
• Latency profiler by @parasj in #3
• Minor cleanup by @parasj in #4
• Bandwidth profiler by @parasj in #5
• Integrate benchmark to profile GCP Premium versus Standard network tiers by @parasj in #6
• Pareto evaluations of optimizer by @parasj in #7
• Gate cloud providers to not load GCP components if not evaluated by @parasj in #8
• Fix AWS and GCP key generation by @parasj in #9
• WIP: Dataplane by @ShishirPatil in #10
• Python dataplane test by @parasj in #11
• Build Python docker image by @parasj in #12
• Benchmark Python dataplane by @parasj in #13
• S3 interface by @parasj in #14
• Pipelining by @parasj in #15
• Initial null replicator (provision only + metadata API) by @parasj in #16
• [gateway] Add new API call to spawn and close server by @parasj in #17
• Add ChunkRequest API by @parasj in #18
• ReplicatorClient example sends ChunkRequests to source gateways by @parasj in #20
• Direct replication with randomly generated objects on source node by @parasj in #21
• Add new pytype workflow, fix some pytype issues by @parasj in #23
• Network graph with traceroute by @sarahwooders in #24
• Replicate across parallel instances by @parasj in #26
• 3-hop overlay works by @parasj in #28
• UDP benchmark by @parasj in #27
• Fix some provisioning issues with GCP and AWS by @parasj in #29
• Some minor fixes to build process by @parasj in #31
• Monitor chunk states using a new API call by @parasj in #30
• Use a single queue as opposed to per-worker queues by @samkumar in #33
• Standardize on gibibyte instead of gigabyte by @parasj in #34
• Provision a new Skylark VPC instead of using the default VPC by @parasj in #35
• Cleaner monitoring loop code w/ pandas by @parasj in #37
• Switch monitoring to netdata by @parasj in #39
• Connection pooling + tune batch size to eliminate stragglers by @parasj in #38
• Maintain a single persistent connection between Gateways by @parasj in #41
• Initial skylark CLI implementation by @parasj in #42
• GatewayReceivers should only close servers when they are done receiving all pending chunks by @ShishirPatil in #43
• Client web UI to display status of transfer by @parasj in #44
• Add past notebooks by @parasj in #46
• Benchmark triangles by @parasj in #45
• skylark replicate_random CLI tool by @parasj in #47
• Benchmark parallel connections by @parasj in #48
• Optionally enable BBR for benchmarks by @parasj in #49
• AWS limits CLI by @parasj in #50
• Dispatch ChunkRequests to source instances in parallel by @parasj in #51
• Send all chunk requests upfront to gateways instead of passing along in the critical path by @parasj in #52
• Benchmark multiple gateways by @parasj in #53
• Check AWS region is enabled before using it by @parasj in #54
• Fix miscellaneous bugs by @samkumar in #56
• Working GCP in gateway by @parasj in #58
• Add support for Microsoft Azure by @samkumar in #55
• Minor README by @ShishirPatil in #61
• Block till sufficient space is available by @parasj in #57
• Fix GCP firewall issue and increases banner timeout for SSH issue by @parasj in #64
• Cache public IPs by @parasj in #67
• [Client] Cancel pending chunk requests by closing gateways by @parasj in #70
• [CLI] Query available Azure SKUs by @parasj in #69
• [CLI] Initialize key cloud authentication details in a config file by @parasj in #65
• Use alternate VMs and disable some regions temporarily by @parasj in #76
• More robust throughput grid profiling by @parasj in #74
• Prompt to increase ulimit for files if it's too low, use fallocate to generate random files by @parasj in #77
• Use updated TCP window by @parasj in #78
• Basic optimizer fixes + optimizer CLI by @parasj in #81
• Fix solver to add connection limit and basic instance limit by @parasj in #82
• Integrate S3Interface with replicator by @sarahwooders in #72
• GCS Object Store Interface by @sarahwooders in #88
• GCSInterface integration with gateway + replicator by @sarahwooders in #91
• Solver instance limit and instance cost by @parasj in #87
• Generate instance-level plans from the solver by @parasj in #92
• Remove web dashboard by @parasj in #93
• Update gateway to run a ReplicationPlan graph by @parasj in #95
• Fix gateway replication by @parasj in #98
• Add GCS support to skylark cp CLI command by @sarahwooders in #97
• Azure Azure Blob Interface by @ShishirPatil in #99
• Migrate from loguru to print by @parasj in #104
• Transfer stall debugging by @parasj in #105
• Azure Gateway and replicator interface by @ShishirPatil in #100
• Exceptions parsing results should not stop throughput grid by @parasj in #110
• Update Dockerfile by @parasj in #111
• Ray autoscaler for Skylark control-plane by @ShishirPatil in #106
• Fix test_replicator_client by @parasj in #115
• CLI integration by @ShishirPatil in #114
• New solver updates for updated Pareto sweep by @parasj in #113
• Performance by @ShishirPatil in #116
• [DRAFT] Fix solver from bad state by @parasj in #119
• [DRAFT] Updated solver dispatch by @parasj in #118
• Work with arbitraty no of gateways by @ShishirPatil in #120
• Integrate skylark replicate-json CLI command with object stores by @sarahwooders in #117
• Process pool for uploading/downloading object store by @sarahwooders in #121
• Full skylark solver sweep by @parasj in #126
• Determine object store keys from listing bucket by @sarahwooders in #127
• Plotti...