Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Virtual keyspace metadata support immplementation #1794

Open
wants to merge 1 commit into
base: trunk
Choose a base branch
from

Conversation

tengu-alt
Copy link
Contributor

@tengu-alt tengu-alt commented Aug 8, 2024

This PR provides access to the virtual metadata tables exposed through the system_virtual_schema keyspace.

Example query:

cqlsh> SELECT * FROM system_virtual_schema.tables;

 keyspace_name         | table_name                       | comment
-----------------------+----------------------------------+----------------------------------------------------
          system_views |                    batch_metrics |               Metrics specific to batch statements
          system_views |                           caches |                                      system caches
          system_views |    cidr_filtering_metrics_counts |           Count metrics specific to CIDR filtering
          system_views | cidr_filtering_metrics_latencies |         Latency metrics specific to CIDR filtering
          system_views |                          clients |                        currently connected clients 

The system_virtual_schema keyspace contains keyspaces, tables, and columns that users can now parse by specifying a keyspace.

Users can retrieve parsed tables using the VirtualKeyspaceMetadata(keyspace string) method for the Session.

The implementation is based on how the driver retrieves regular keyspace metadata.

@tengu-alt tengu-alt force-pushed the virtual-table-metadata-support branch 3 times, most recently from 558eec1 to 5bcb70b Compare August 8, 2024 11:18
metadata.go Outdated
Comment on lines 884 to 889
const stmt = `
SELECT
table_name,
comment
FROM system_virtual_schema.tables
WHERE keyspace_name = ?`

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please fix indent:

Suggested change
const stmt = `
SELECT
table_name,
comment
FROM system_virtual_schema.tables
WHERE keyspace_name = ?`
const stmt = `
SELECT
table_name,
comment
FROM system_virtual_schema.tables
WHERE keyspace_name = ?`

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agree, fixed

metadata.go Outdated
Comment on lines 1102 to 1110
const stmt = `
SELECT
table_name,
column_name,
clustering_order,
kind,
type
FROM system_virtual_schema.columns
WHERE keyspace_name = ?`

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same ident problem:

Suggested change
const stmt = `
SELECT
table_name,
column_name,
clustering_order,
kind,
type
FROM system_virtual_schema.columns
WHERE keyspace_name = ?`
const stmt = `
SELECT
table_name,
column_name,
clustering_order,
kind,
type
FROM system_virtual_schema.columns
WHERE keyspace_name = ?`

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

func (s *virtualSchemaDescriber) getSchema(keyspaceName string) (*VirtualKeyspaceMetadata, error) {
s.mu.Lock()
defer s.mu.Unlock()
metadata, found := s.cache[keyspaceName]
Copy link

@dkropachev dkropachev Aug 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't you invalidate it at some point.
Say you upgrade cluster from one version to another, driver will still keep old schema.

Probably on handleNodeUp you could read release_version select release_version from system.local where key = 'local' and invalidate or update cache if it has changed.

Another problem if your cluster half on new version half on another.
Now you depends on where have your control connection landed, if it has landed on a node with new version, session will give you a new schema, while old nodes don't support it.
Way to solve it is to run queries not on control connection, but rather on the node with oldest version available, if you were do so, you can't pick all the nodes from the cluster, you have to adhere to HostSelectionPolicy, say if it is dcAwareRR you will need to filter out all nodes from other datacenters.

Copy link
Contributor Author

@tengu-alt tengu-alt Aug 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with your point about invalidation, but virtual metadata is only requested on the user's purpose, not by some node event, also virtual tables can't be changed, if we talk about some schemaAgreement case. In my opinion, it is unnecessary to invalidate it on every getSchema request, and i think it is unnecessary to remove a control connection, it is good to keep the existing pattern of the metadata requests.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you clear cache a the betting of getSchema it means that there is going to be no cache, why just not drop it.
If you drop the cache then there is no point in having virtualSchemaDescriber, you can just move code to Session.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I totally agree with you, but in my opinion, it is good to keep an existing pattern of processing the metadata (virtual or not)

Comment on lines +268 to +269
mu sync.Mutex
cache map[string]*VirtualKeyspaceMetadata

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would recommend, in this particular case, use sync.Map instead of Mutex + map.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is preferable to use mutex, to keep the existing pattern, and avoid unnecessary code complexity.

@tengu-alt tengu-alt force-pushed the virtual-table-metadata-support branch from 5bcb70b to 4b78420 Compare August 12, 2024 12:21
@tengu-alt tengu-alt changed the title Virtual keyspace metadata support was immplemented Virtual keyspace metadata support immplementation Nov 18, 2024
@tengu-alt tengu-alt force-pushed the virtual-table-metadata-support branch from 4b78420 to 4c3aefe Compare November 20, 2024 11:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants