V1.0.0

Release v1.0.0

Release Date: 19th January 2022
Codename: New Horizons

Summary

With this release Mayastor is considered graduated from pre-release / beta status.

Features

• Complete reimplementation of the control plane, focused on modularity, future extensibility and scaling

  • Now written in Rust
  • Public RESTful API
  • k8s-like reconciliation loop behavior for Mayastor managed objects (volumes, pools)

• Extensive bug fixing and stabilization

  • Supported by 1000's of hours of automated system testing

• New Mayastor kubectl plugin for management and monitoring

Testing

This release has been subject to E2E system testing by MayaData / DataCore Software, running under Ubuntu 20.04.3_LTS (kernel: ubuntu-5.8.0.63-generic)

• Tested k8s versions
  • 1.20.9
  • 1.21.8
  • 1.22.5
  • 1.23.1

Known Issues

• A Mayastor container may fault and restart if a disk device used by one of its associated disk pools becomes inaccessible. This is currently under investigation, with the intention of providing a fix in the next release.

• Deploying an application pod on a worker node which hosts both Mayastor and Prometheus exporter causes that node to restart.

  • workaround: Use kernel version extra-5.31.0 or later

Getting Started

Mayastor user documentation, including a quick deployment guide, can be found here

Upgrade

Upgrades from previous versions of Mayastor are not supported. Any earlier release should be removed prior to installing this version. It is intended that releases forward of v1.0.0 will allow in-place upgrades.

Support

If you are having issues during installation, configuration or upgrade, you can contact us via:

• OpenEBS on Kubernetes Slack community
  • Already signed up? Head to our discussions at #openebs
• Raising an issue

v0.8.1-beta

Various stability fixes

v 0.8.0-beta

Release v0.8.0

Release Date: 15th March 2021
Codename: March Hare

Summary

Incremental/Patch Release

OpenEBS Mayastor is considered beta software. Please deploy accordingly.

Changelog (Highlights)

Features

feat: build against SPDK 21.01
feat: unify mayastor-client address/port flags
feat(nexus): allow shared replica to be added as local child
feat: make bdev share protocol argument as an option
feat: make bdev create parameters strict
feat(nexus): implement set ANA state for an NVMf-published Nexus
feat(nexus): implement get ANA state for an NVMf-published Nexus
feat: show kernel nvme initiator multipath status on startup
docs: add requirements for running with ANA

Fixes

fix(mayastor): fallible replica share (CAS-714)
fix: correct dead links in build docs
fix: update CSI sidecars used with mayastor CAS-651
fix: correct safety of maya_log
fix: try to log the full spdk msg
fix: mayastor-client nexus create accept many children
fix(moac): mayastor pool state is shown as blank
fix: maintain consistency with other args
fix(nexus): mayastor hangs when creating a nexus if there is an error
fix(composer): update separator for security options
fix(csi): update CSI external provisioner to 2.1.1

Chores

refactor(json): tweak the jsongrpc methods
refactor(deployer): split into library and binary
chore(deps): bump systeminformation
chore(nightly): update nix rust nightly
revert: remove rust-toolchain
refactor(nexus): clean up label parsing code
chore: upgrade bindgen to 0.57
chore: do profile-based yaml generation
chore(operators): purge rust operator from repo
chore: finangle multi ISA builds
chore: re-enable on x86_64
chore: compat finangling
chore: update docs
chore: fix cli tests to use -b
chore: clean up and refine spdk use
chore: touchup log wrapper lints and comments
chore: formatting nits
refactor(ctrlp): add core agent
chore: fixup core_6 test failure
chore: remove control plane from mayastor
test(cargo): verify unmap operation for aio module
test: uring add and removal
test(local): add compose override flags

Experimental (Work in progress, unsupported, exposed interfaces may be brittle)

feat(rest): expose list_block_devices method
feat(rest client): add support for new rest calls
feat(rest): authenticate clients
feat(rest-test): new rest test project
chore(rest): add openapi spec to the tree
refactor(rest_cli): improve rest client errors
fix(ci): openapi-check should fail
fix(openapi): parameter names should match
test(rest): add more tests using rest

feat: enable nix-shell on aarch64
feat: enable cargo build on aarch64
feat: aarch64 passing tests
chore: try to enable cross builds
chore: delay cross builds

Testing

This release has been subject to E2E testing by MayaData. Worker nodes ran Ubuntu 20.04.2 LTS, using the docker runtime 20.10.5 under Kubernetes version 1.19.8

Known Issues

Mayastor may fault during the retirement of a nexus child. This is currently under investigation.

Getting Started

Mayastor user documentation, including a quick deployment guide, can be found here

Upgrade

Upgrades from previous versions of Mayastor are not supported. Any earlier release should be removed prior to installing this version. For testing purposes, it is preferable to install to a new, clean cluster.

Support

If you are having issues during installation, configuration or upgrade, you can contact us via:

• OpenEBS on Kubernetes Slack community
  • Already signed up? Head to our discussions at #openebs
• Raising an issue

v 0.7.1-beta

Release v0.7.1

Release Date: 12th February 2021
Codename: lunar oxen

Summary

Patch Release

OpenEBS Mayastor is considered beta software. Please deploy accordingly.

Changelog (Highlights)

Fixes

fix(nvmeadm): remove unwrap from disconnect
fix(csi): remove csi-driver-registrar prestop hook
fix(moac): serialize volume creation
fix(nexus): allow re-adding child removed by AEN
fix(nexus): replace NexusChild destroying flag with child state
fix(nexus): set children to destroying state on shutdown
fix(moac): sync replicas more often

refactor(moac): drop support for NBD in CSI
refactor(nexus): add nexus create log
refactor(ctrl-plane): tidy up the control-plane
refactor(nix): control plane nix derivations

Documentation

chore: add CONTRIBUTING.md
feat: rework build docs

Experimental (Work in progress, unsupported)

feat(openapi): generate openapi spec for rest
feat(swagger_ui): expose swagger-ui on rest
feat(deployer): new tool to test the control plane
feat(deployer): ability to deploy JsonGrpc service
feat(rest): expose json rpc methods
feat(control plane): allow images to be built
refactor(rest): add cmdline certificate arguments
refactor(rest): improve rest versions

Getting Started

Mayastor user documentation, including a quick deployment guide, can be found here

Upgrade

Upgrades from previous versions of Mayastor are not supported. Any earlier release should be removed prior to installing this version. For testing purposes, it is preferable to install to a new, clean cluster.

Support

If you are having issues during installation, configuration or upgrade, you can contact us via:

• OpenEBS on Kubernetes Slack community
  • Already signed up? Head to our discussions at #openebs
• Raising an issue

v 0.7.0-beta

Release v0.7.0

Release Date: 25th January 2021
Codename: timorous beastie

Summary

Maintenance Release

OpenEBS Mayastor is considered beta software. Please deploy accordingly.

Changelog

To follow (work in progress)

Getting Started

Mayastor user documentation, including a quick deployment guide, can be found here

Upgrade

Upgrades from previous versions of Mayastor are not supported. Any earlier release should be removed prior to installing this version. For testing purposes, it is preferable to install to a new, clean cluster.

Support

If you are having issues during installation, configuration or upgrade, you can contact us via:

• OpenEBS on Kubernetes Slack community
  • Already signed up? Head to our discussions at #openebs
• Raising an issue

v 0.6.0-beta

Release v0.6.0

Release Date: 1st December 2020

Summary Changelog

Maintenance Release

Project Status Upgraded: BETA PHASE
With this release, OpenEBS Mayastor is considered beta software. Please deploy accordingly.

Getting Started

Mayastor user documentation, including a quick deployment guide, can be found here

Upgrade

Upgrades from previous versions of Mayastor are not supported. Any earlier release should be removed prior to installing this version. For testing purposes, it is preferable to install to a new, clean cluster.

Support

If you are having issues during installation, configuration or upgrade, you can contact us via:

• OpenEBS on Kubernetes Slack community
  • Already signed up? Head to our discussions at #openebs
• Raising an issue

v 0.5.0-alpha

Release Summary

Release Date: 14th October 2020

Mayastor is currently in an early stage of development. This release is considered alpha software. As such, it is not intended for use in production.

New Features:

• Block Device Discovery - Mayastor instances now expose a gRPC API which is used to enumerate block disk devices attached to the host node, as an aid to the identification of suitable candidates for inclusion within storage Pools during configuration. This functionality is also accessible within the mayastor-client diagnostic utility.

Enhancements:

• Storage Pool Finalizers - Storage pools are now protected from inadvertent deletion whilst they contain replicas, through the addition of a finalizer to the Pool CRD object by the pool operator when the pool replica count is non-zero.

Bug Fixes:

9 new bugs were identified during testing.
5 bugs closed, resolution "fixed".

As of 15th September 2020, all bugs found by MayaData during internal testing and development are being logged, triaged and resolved directly within the Mayastor GitHub repository, thereby granting visibility to the OpenEBS and Mayastor communities.

Getting Started

The Mayastor quick deployment guide can be found here

Upgrade

Upgrades from previous versions of Mayastor are not supported. Any earlier release should be removed prior to installing this version. For testing purposes, it is preferable to install to a new, clean cluster.

Known Issues

Issue: Some Mayastor custom resource objects may lack status information after creation

Symptom
After creation, the status section of some custom resource objects may be seen to be missing. For example, in the case of a newly declared Mayastor Pool (MSP):

kubectl get -n mayastor get msp
NAME         NODE                                STATE   AGE
diskpool-1   aks-agentpool-41030923-vmss000003           46m

This has also been seen to affect Mayastor Volume (MSV) custom resource objects, which similarly may lack status information after creation and may be unavailable for use as a result.

Workaround
Restart the Mayastor MOAC pod. This can be achieved by deleting the pod, causing k8s to reschedule it in accordance with the moac deployment definition. Affected state information should be restored once MOAC has restarted.

Status
Confirmed bug. A fix for this issue is being targeted for the scheduled 0.6.0 release.

Support

If you are having issues during installation, configuration or upgrade, you can contact us via:

• OpenEBS on Kubernetes Slack community
  • Already signed up? Head to our discussions at #openebs
• Raising an issue

v 0.4.0-alpha

Release Summary

Release Date: 18th September 2020

Mayastor is currently in an early stage of development. This release is considered alpha software. As such, it is not intended for use in production.

New Features:

• Enhanced User Documentation - The previous user quickstart guide has been updated and extended, and is now hosted on GitBook. This new content deprecates the README.md file in the /deploy directory of the Mayastor repository. GitBook will be the principle repository of Mayastor user documentation during this phase of development.
• k3s Support - Mayastor may be deployed on kubernetes clusters built using the lightweight k3s distribution

Enhancements:

• Rebuild Restart Tolerance - If one or more children of a Nexus are either pending, or undergoing, rebuild this process this will now resume automatically upon a restart of the Mayastor pod hosting that nexus. Child state will be correctly deduced on restart.
• Nexus I/O Internal Retry - In the event that an I/O operation directed at the child of a nexus fails, the nexus will internally queue and retry (attempt count is configurable) instead of immediately returning an error to the consuming application container via the front-end target. This isolates the application from the effects of transient events e.g. temporary losses of connectivity.
• Pool Auto-naming - As a convenience to the user, if no value is specified in the name field of a Mayastor Pool (MSP) declaration, a unique name will be generated automatically during pool creation.

Bug Fixes:

14 new bugs were identified during testing.
16 bugs closed, resolution "fixed".

From 15th September 2020, all bugs found by MayaData during internal testing and development will be logged, triaged and resolved directly within the Mayastor GitHub repository, thereby granting visibility to the OpenEBS and Mayastor communities.

Getting Started

The Mayastor quick deployment guide can be found here

Upgrade

Upgrades from previous versions of Mayastor are not supported. Any earlier release should be removed prior to installing this version. For testing purposes, it is preferable to install to a new, clean cluster.

Known Issues

Issue: Mayastor Pool (MSP) custom resource objects may lack status information after creation

Symptom
After creating a new pool definition the status section of the MayastorPool custom resource is missing. For example:

kubectl get -n mayastor get msp
NAME         NODE                                STATE   AGE
diskpool-1   aks-agentpool-41030923-vmss000003           46m

Workaround
Restart the Mayastor MOAC pod. This can be achieved by deleting the pod, causing k8s to reschedule it in accordance with the moac deployment definition. Pool state information should be available once MOAC has restarted.

Status
Confirmed bug

Support

If you are having issues during installation, configuration or upgrade, you can contact us via:

• OpenEBS on Kubernetes Slack community
  • Already signed up? Head to our discussions at #openebs
• Raising an issue

v 0.3.0-alpha

Release Summary

Release Date: 12th August 2020

Mayastor is currently in an early stage of development. This release is considered alpha software. As such, it is not intended for use in production.

New Features:

• Automatic Replica Replacement - A Replica which has entered the "Faulted" state will be automatically replaced by the control plane, subject to there being available spare capacity in an appropriate Pool. The content of the replacement Replica will be 'rebuilt', i.e. made consistent with the other Replicas of that Nexus. This occurs without significant impact upon workload I/O.
• NVMe-oF Support - The Mayastor CSI plugin can provision Persistent Volumes which are mounted on worker nodes over NVMe-oF, by specifying this transport type option in the corresponding StorageClass. (This functionality is made conditional upon kernel mode NVMe-oF initiator support, necessitating the use of kernel version 5.1 or greater).
• Block Mode - Persistent Volumes now support raw block access: volumeMode: block
• Direct Device Access - Mayastor Pools now support direct (host kernel independent) access to block device members, reducing abstraction overhead for high performance devices. The Mayastor Pool CRD accepts the previous form e.g. disks: ["/dev/sdx"], in addition to a new URI format e.g. disks: ["nvmf://10.240.0.4:8420/...etc"]

Enhancements:

• Plugin Separation - The Mayastor CSI driver and plugins have been moved from within the Mayastor DaemonSet to their own, dedicated DaemonSet. This allows them to be independently scheduled by k8s and as a result, Mayastor Persistent Volumes can now be mounted and consumed on any worker node in the cluster, not just on those which are also running an instance of the Mayastor storage engine.
• Rebuild Performance - The previous limitation on the maximum outstanding I/O permitted by a rebuild process has been raised, allowing faster Replica rebuilds but without significant impact on workload I/O to the affected Nexus. Each rebuild now supports up to 16 concurrent copy tasks, with the limitation on total rebuild I/O outstanding increased to 1MiB.
• Durability - Mayastor pods persist their configuration state to local host storage each time they they process an API call which results in a state change. This allows a Mayastor pod to be correctly restarted in the event that it or its host node stop unexpectedly. At this time, such a restart will permit access to Replicas shared with other Nexus' but will not restore access to workload I/O for any PVC exported by the affected Mayastor process. Automated Failover and attendant continuous access to data whilst in a degraded state is the subject of ongoing development and will be introduced in future releases.
• SPDK - Mayastor now uses SPDK version 20.07
• Replica Error Threshold - A Nexus maintains counts of I/O error responses arising from its child bdevs ('Replicas'). In this version, if the error count held for a Replica exceeds a (default) value of 64, that Replica will be marked as "Faulted" by the Nexus and will be removed from the I/O path. In this release, the threshold value may only be changed by recompilation.
• Flush - The Nexus now implements support for the flush command

Getting Started

The Mayastor quick deployment guide can be found here

Upgrade

Upgrades from previous versions of Mayastor are not supported. Any earlier release should be removed prior to installing this version. For testing purposes, it is preferable to install to a new, clean cluster.

Support

If you are having issues in setting up or upgrade, you can contact us via:

• OpenEBS on Kubernetes Slack community
  • Already signed up? Head to our discussions at #openebs
• Raise an issue

v 0.2.0-alpha

Release Summary

Release Date: 15th June 2020

Mayastor is currently in an early stage of development. This release is considered alpha software. As such, it is not intended for use in production.

New Features:

• Rebuild Process - A rebuild mechanism has been added to Mayastor which, through the scheduling of multiple concurrent rebuild 'tasks', is able to bring a new child (i.e. replica) which is being added to an existing Nexus into a consistent state with its current children and into synchronisation with the Nexus' front-end volume (i.e. Persistent Volume), thereby increasing/restoring workload protection levels. It does so without disrupting workload IO. (Known Limitation: Any IO error encountered either at the source or destination child of a rebuild task will cause the rebuild process to halt and no attempt will be made to retry. This will be addressed in a future release).
• NVMe-oF Support - A Nexus can be created and shared (i.e. a front-end target established) using the NVMe protocol. (Known Limitation: It is not currently possible to provision an NVMe-oF Nexus via CSI. This functionality will be introduced in a future release).
• Mayastor Node (msn) CRD - Introduced for the persistence and observabilty of Mayastor-enabled cluster node configuration and membership
• Mayastor-client - A gRPC-based diagnostic and low-level configuration tool for Mayastor is introduced as a planned successor to the existing mctl utility. It is expected that mctl will be deprecated in a future release.

Enhancements:

• The data path now categorises and maintains counts of error IO responses received from child bdevs. This is to allow the development of automated faulting and recovery processes in future releases.
• The size of the MOAC container image has been reduced. Previously this ran to multiple hundreds of MBs.
• MOAC and Mayastor-test have been refactored to make use of v12 node.js, from v10 previously. This brings a number of improvements including; faster startup performance, heap size improvements and a more secure implementation of TLS.
• The Nexus now correctly handles reset IOs sent by workload initiators. Resets are propagated to each child of the nexus and the appropriate response returned to the initiator.
• The Mayastor Volume (msv) CRD now includes the rebuild and health status of its corresponding children

Bug Fixes:

• Issue: When creating a nexus with multiple children, if the second or any subsequent child were inaccessible a panic was encountered
  • Resolution: During creation, first close any child bdevs which may already be open. Abort the nexus creation if an error is encountered during this step.

Getting Started

The Mayastor quick deployment guide can be found here

Upgrade

Upgrades from previous versions of Mayastor are not supported. Any earlier release should be removed prior to installing this version. For testing purposes, it is preferable to install to a new, clean cluster.

Support

If you are having issues in setting up or upgrade, you can contact us via:

• OpenEBS on CNCF Slack community
  • Already signed up? Head to our discussions at #openebs
• OpenEBS on Kubernetes Slack community
  • Already signed up? Head to our discussions at #openebs
• Raise an issue