Version 0.9.0

Vulnerabilities

• Fixed a vulnerability that made it possible for a malicious plugin archive that contained entry names with path traversal elements to execute arbitrary code on the filesystem, if installed. This vulnerability, which was first disclosed by the Snyk Security Research Team, existed because Concourse did not verify that an entry, potentially extracted from a zipfile, would exist within the target directory if actually extracted. We've fixed this vulnerability by switching to the zt-zip library for internal zip handling. In addition to having protections against this vulnerability, zt-zip is battle-tested and well maintained by ZeroTurnaround. Thanks again to the Snyk Security Research Team for disclosing this vulnerability.

Security Model

• Added a notion of user roles. Each user account can either have the ADMIN or USER role. ADMIN users are permitted to invoke management functions whereas accounts with the USER role are not.
  • All previously existing users are assigned the ADMIN role on upgrade. You can change a user's role using the users CLI.
  • The users create command now requires a role to be provided interactively when prompted or non-interactively using the --set-role parameter.
• Added an edit option to the users CLI that allows for setting a user's role and/or changing the password. The password can also still be changed using the password option of the users CLI.
• Removed a constraint the prevented the default admin user account from being deleted.
• Added additional logging around the upgrade process.
• Fixed a bug that prevented upgrade tasks from being run when upgrading a Concourse Server instance that was never started prior to the upgrade.
• Upgraded some internal libraries to help make server startup time faster.
• Fixed a bug in concourse-driver-java that caused the navigate functions to report errors incorrectly.
• Added user permissions. Each non-admin user account can be granted permission to READ or WRITE data within a specific environment:
  • Permissions can be granted and revoked for a non-admin role user by a user who has the admin role.
  • Permissions are granted on a per environment basis.
  • A user with READ permission can read data from an environment but cannot write data.
  • A user with WRITE permission can read and write data in an environment.
  • Users with the admin role implicitly have WRITE permission to every environment.
  • If a user's role is downgraded from admin to user, she will have the permissions she has before being assigned the admin role.
  • If a user attempts to invoke a function for which she doesn't have permission, a PermissionException will be thrown, but the user's session will not terminate.
  • A user with the admin role cannot have any of her permissions revoked.
  • Plugins automatically inherit a user's access (based on role and permission).
  • Service users that operate on behalf of plugins have WRITE access to every environment.

Data Types

• Added a Criteria#at(Timestamp) method to transform any Criteria object into one that has all clauses pinned to a specific Timestamp.
• Added a static Criteria#parse(String) method to parse a CCL statement and produce an analogous Criteria object.
• Streamlined the logic for server-side atomic operations to unlock higher performance potential.
• Added short-circuit evaluation logic to the query parsing pipeline to improve performance.
• Added a TIMESTAMP data type which makes it possible to store temporal values in Concourse.
  • The concourse-driver-java API uses the Timestamp class to represent TIMESTAMP values. Please note that hallow Timestamps (e.g. those created using the Timestamp#fromString method cannot be stored as values). An attempt to do so will throw an UnsupportedOperationException.
  • The concourse-driver-php uses the DateTime class to represent TIMESTAMP values.
  • The concourse-driver-python uses the datetime class to represent TIMESTAMP values.
  • The concourse-driver-ruby uses the DateTime class to represent TIMESTAMP values.
  • The Concourse REST API allows specifying TIMESTAMP values as strings by prepending and appending a | to the value (e.g. |December 30, 1987|). It is also possible to specify a formatting pattern after the value like |December 30, 1987|MMM dd, yyyy|.
• Added a Timestamp#isDateOnly method that returns true if a Timestamp does not contain a relevant temporal component (e.g. the Timestamp was created from a date string instead of a datetime string or a timestring).

Performance

• Upgraded the CCL parser to a newer and more efficient version. This change will yield general performance improvements in methods that parse CCL statements during evaluation.

Developer Experience

• The test Concourse instance used in a ClientServerTest will no longer be automatically deleted when the test fails. This will allow for manual inspection of the instance when debugging the test failure.
• Added additional logging for plugin errors.
• Added a manage interface to the driver APIs. This interface exposes a limited number of management methods that can be invoked programatically.

Bug Fixes

• Fixed a bug that caused the server to fail to start if the conf/stopwords.txt configuration file did not exist.
• Fixed a bug that caused PrettyLinkedHashMap#toString to render improperly if data was added using the putAll method.
• Fixed a bug in the ConcourseImportDryRun#dump method that caused the method to return an invalid JSON string.
• Fixed a bug where a users whose access had been disabled was automatically re-enabled if her password was changed.

Miscellaneous

• Added the ability for the storage engine to track stats and metadata about database structures.

Version 0.8.2

• Fixed a bug in the ManagedConcourseServer#install method that caused the server installation to randomly fail due to race conditions. This caused unit tests that extended the concourse-ete-test-core framework to intermittently fail.

Version 0.8.1

• Fixed a bug that caused local CCL resolution to not work in the findOrInsert methods.
• Fixed an issue that caused conversion from string to Operator to be case sensitive.
• Fixed a bug that caused the putAll method in the map returned from TrackingMultimap#invert to store data inconsistently.
• Added better error handling for cases when an attempt is made to read with a value with a type that is not available in the client's version.
• Fixed a bug that caused Concourse Server to unreliably stream data when multiple real-time plugins were installed.
• Fixed a bug that caused Concourse Server to frequently cause high CPU usage when multiple real-time plugins were installed.
• Added an isolation feature to the ImportDryRunConcourse client (from the concourse-import framework). This feature allows the client to import data into an isolated store instead of one shared among all instances. This functionality is not exposed to the import CLI (because it isn't necessary), but can be benefical to applications that use the dry-run client to programmatically preview how data will be imported into Concourse.
• Added an implementation for the ImportDryRunConcourse#describe method.

Version 0.8.0

• Added a count aggregation function that returns the number of values stored
  • across a key,
  • for a key in a record, or
  • for a key in multiple records.
• Added a max aggregation function that returns the largest numeric value stored
  • across a key,
  • for a key in a record, or
  • for a key in multiple records.
• Added a min aggregation function that returns the smallest numeric value stored
  • across a key,
  • for a key in a record, or
  • for a key in multiple records.
• Moved the ccl parsing logic into a separate library to make the process portable to plugins and other applications.
• Fixed some bugs that could have caused incorrect evaluation of select(criteria), find(criteria) and related methods in some cases.
• Added a TObject#is(operator, values...) method so plugins can perform local operator based comparisons for values returned from the server.

Version 0.7.3

• Fixed a bug that caused the temporal average and sum calculations to fail if the timestamp parameter was generated from a String instead of long.
• Fixed a couple of bugs that made it possible for Concourse Server to pass blank or unsanitized environment names to plugins during method invocations.
• Fixed a bug that caused Criteria objects to be improperly serialized/deserialized when passed to plugin methods as arguments or used as return values.

Version 0.7.2

• Added more detailed information to the server and plugin log files about plugin errors.
• Fixed a bug where TrackingMultimap#percentKeyDataType returned NaN instead of 0 when the map was empty.
• Added a memoryStorage option to the PluginStateContainer class.

Version 0.7.1

• Fixed a bug that caused an error in some cases of importing or inserting data that contained a value of -.
• Added better error message for TApplicationException in CaSH.

Version 0.7.0

• Added navigate methods that allow selecting data based on link traversal. For example, it is possible to select the names of the friends of record 1's friends by doing

    navigate "friends.friends.name", 1
  

• Re-implemented the users CLI to provide extensible commands. Now the users CLI will respond to:

  1. create - create a new user
  2. delete - delete an existing user
  3. enable - restore access to a suspended user
  4. password - change a user's password
  5. sessions - list the current user sessions
  6. suspend - revoke access for a user

• Changed the envtool CLI to the environments CLI with extensible commands. The environments CLI will respond:

  1. list - list the Concourse Server environments

• Changed the dumptool CLI to the data CLI with extensible commands. The data CLI will respond to:

  1. dump - dump the contents of a Concourse Server data file
  2. list - list the Concourse Server data files

• Added a CompositeTransformer to the concourse-import framework that invokes multiple transformers in declaration order.

• Added a Transformers utility class to the concourse-import framework API.

• Fixed a bug that caused the loss of order in plugin results that contained a sorted map.

• Added a --dry-run flag to the import CLI that will perform a test import of data in-memory and print a JSON dump of what data would be inserted into Concourse.

• Added support for installing multiple plugins in the same directory using the concourse plugin install </path/to/directory> command.

• Implemented describe() and describe(time) methods to return all the keys across all records in the database.

• Fixed a bug where the browse(keys, timestamp) functionality would return data from the present state instead of the historical snapshot.

• Fixed an issue that caused plugins to use excessive CPU resources when watching the liveliness of the host Concourse Server process.

• Added a bug fix that prevents service tokens from auto-expiring.

• Added a ps command to the plugins CLI to display information about the running plugins.

• Fixed a bug that caused the average(key) method to return the incorrect result.

• Fixed a bug that caused calculations that internally performed division to prematurely round and produce in-precise results.

• Fixed a bug that caused the editing and deleting an existing user with the users CLI to always fail.

• Added support for defining custom importers in .jar files.

• Detect when the service is installed in an invalid directory and fail appropriately.

• Fixed a security bug that allowed the invokePlugin method to not enforce access controls properly.

• Fixed a bug that caused management CLIs to appear to fail when they actually succeeded.

• Improved the performance of the ResultDataSet#put method.

• Fixed a bug in the implementation of ObjectResultDataset#count.

• Deprecated Numbers#isEqual and Numbers#isEqualCastSafe in favor of better names Numbers#areEqual and Numbers#areEqualCastSafe.

• Added support for getting the min and max keys from a TrackingMultimap.

• Added an ImmutableTrackingMultimap class.

• Fixed a bug in the TrackingMultimap#delete method.

• Fixed the CPU efficiency of the JavaApp host termination watcher.

• Fix bug that caused JavaApp processes to hang if they ended before the host was terminated.

• Added database-wide describe method.

Version 0.6.0

• Added calculate interface to the java driver to perform aggregations.
• Added a sum aggregation function.
• Added an average aggregation function.
• Switched to socket-based (instead of shared memory based) interprocess communication between Concourse Server and plugins.
• Assigned meaningful process names to plugins.
• Added a System-Id for each Concourse Server instance.
• Fixed bugs in the ObjectResultDataset implementation.
• Added an end-to-end testing framework for the plugin framework.
• Fixed a bug that caused some query results to be case-sensitive.
• Fixed a bug that caused some query results to have inconsistent ordering.
• Upgraded support for parsing natural language timestamps.
• Updated the usage method of the concourse init.d script.
• Fixed a bug that caused PluginContext and PluginRuntime to return different directories for a plugin's data store.
• Added a progress bar for the plugin install command.
• Fixed a bug that caused ConcourseRuntime to mishandle plugin results.
• Clarified the proper way to use plugin hooks.
• Refactored the plugin management CLI.
• Fixed a bug that allowed plugins to invoke server-side transaction methods (CON-518).
• Refactored the implementation of the version CLI.
• Improved process forking framework.
• Enabled console logging for plugins (CON-514).
• Made the Transformer interface in concourse-import framework a FunctionalInterface.
• Added logic to plugins to signal to Concourse Server when initialization has completed.
• Added functionality to get the host Concourse Server directory from the import CLI and server-side management CLIs.
• Added support for defining custom importers in an importers directory within the Concourse Server instance directory.
• Added a --annotate-data-source option to the import CLI that will cause imported records to have the name of the source file added to the __datasource key.
• Added support for specifying the id of the record into which data should be inserted within the JSON blob that is passed to the insert method.
• Added method to TrackingMultimap that measures the spread/dispersion of the contained data.
• Fixed a race condition bug in the concourse-ete-test framework.
• Fixed bug that caused a preference for using random ports outside the ephemeral range.
• Changed the plugin configuration to no longer require setting remote_debugger = on to enable remote debugging; now it is sufficient to just specify the remote_debugger_port preference.

Version 0.4.4

• Fixed an issue where transactions and atomic operations unnecessarily performed pre-commit locking during read operations, which negatively impacted performance and violated the just-in-time locking protocol (CON-198/CON-199).
• Added logic to prevent the Buffer from attempting a scan for historical data that is older than any data that is currently within the Buffer (CON-197).
• Added group sync: an optimization that improves Transaction performance by durably fsyncing committed writes to the Buffer in bulk. Transactions still honor the durability guarantee by taking a full backup prior to acknowledging a successful commit (CON-125).
• Improved the performance of releasing locks by moving garbage collection of unused locks to a background thread.
• Improved the performance for upgrading range locks and checking for range conflicts by using collections that shard and sort range tokens.
• Improved Transaction write performance by using local bloom filters to speed up verifies.
• Fixed a bug where storage engine methods that touched an entire record (e.g. browse(record) and audit(record)) or an entire key (browse(key)) were not properly locked which potentially made reads inconsistent (CON-239).
• Fixed an issue where transactions unnecessarily performed double write validation which hurt performance (CON-246).
• Fixed a major memory leak that occurred when transactions were aborted or failed prior to committing (CON-248).
• Added logging to indicate if the background indexing job terminates because of an uncaught error (CON-238).
• Fixed an issue where the background indexing job could be wrongfully terminated because it appeared to be stalled when doing a large amount of work.
• Fixed a memory-leak issue where Concourse Server did not release resources for abandoned transactions if the client started a transaction and eventually started another one without explicitly committing or aborting the previous one (CON-217).
• Fixed various issues and performance bottlenecks with syncing storage blocks to disk.
• Improved the names of several Concourse Server threads.