Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PSQL changes & naksha alignment #102

Merged
merged 19 commits into from
Nov 28, 2023
Merged

PSQL changes & naksha alignment #102

merged 19 commits into from
Nov 28, 2023

Conversation

Amaneusz
Copy link
Collaborator

No description provided.

@Amaneusz Amaneusz marked this pull request as draft November 27, 2023 10:13
@Amaneusz Amaneusz marked this pull request as ready for review November 27, 2023 15:26
Copy link

Code Coverage

Overall Project 31.39% -14.11% 🍏
Files changed 25.59% 🍏

Module Coverage
here-naksha-lib-heapcache 53.36% -3.89% 🍏
here-naksha-app-service 50.34% -0.77% 🍏
here-naksha-lib-hub 35.35% -14.65% 🍏
here-naksha-lib-core 30.04% -9% 🍏
here-naksha-lib-psql 20.73% -48.62% 🍏
Files
Module File Coverage
here-naksha-lib-heapcache HeapCache.java 79.06% 🍏
HeapFeatureWriter.java 50% -24.53% 🍏
here-naksha-app-service EventHandlerApiTask.java 93.87% 🍏
StorageApiTask.java 93.87% 🍏
SpaceApiTask.java 93.87% 🍏
WriteFeatureApiTask.java 82.65% 🍏
AccessLogUtil.java 81.47% 🍏
AbstractApiTask.java 73.57% 🍏
ReadFeatureApiTask.java 72.83% 🍏
NakshaApp.java 64.58% 🍏
NakshaHttpVerticle.java 55.28% -3.85% 🍏
StorageMaintainer.java 0% 🍏
here-naksha-lib-hub MockResult.java 100% 🍏
NHSpaceStorage.java 79.72% 🍏
NakshaHubFactory.java 75.61% 🍏
NHSpaceStorageWriter.java 67.65% -1.47% 🍏
NHAdminMock.java 58.2% -12.17% 🍏
NHSpaceStorageReader.java 53.33% 🍏
MockResultCursor.java 48.46% -24.62% 🍏
NHAdminWriterMock.java 33.9% -32.77% 🍏
NakshaHubMock.java 26.36% -26.36% 🍏
NHAdminReaderMock.java 3.25% -28.05% 🍏
NakshaHub.java 0% -15.5% 🍏
ConfigUtil.java 0% -15% 🍏
NHAdminStorage.java 0% 🍏
NHAdminStorageWriter.java 0% -30.95% 🍏
here-naksha-lib-core JsonEnumSerializer.java 100% 🍏
XyzError.java 92.54% -7.46% 🍏
Json.java 89.86% -1.09% 🍏
JsonModule.java 86.92% -3.74% 🍏
FibLinearProbeTable.java 86.52% 🍏
FibMapIterator.java 83.33% 🍏
ILike.java 82.93% -12.2% 🍏
JsonFieldByte.java 75.46% 🍏
FibRefType.java 74.19% 🍏
JsonFieldShort.java 73.77% 🍏
JsonFieldBool.java 72.11% 🍏
JsonConfigFile.java 71.34% -2.59% 🍏
JsonFieldDouble.java 71.03% 🍏
JsonFieldFloat.java 71.03% 🍏
JsonFieldInt.java 69.9% 🍏
JsonFieldLong.java 69.31% 🍏
JsonEnum.java 67.72% -7.61% 🍏
FibMap.java 64.81% -3.5% 🍏
JsonEnumDeserializer.java 63.04% 🍏
JsonFieldChar.java 62.03% -3.21% 🍏
FibSet.java 56.37% -2.49% 🍏
JsonSerializable.java 55.36% -2.77% 🍏
AbstractTask.java 54.75% -17.51% 🍏
JTSHelper.java 54.31% 🍏
JsonObject.java 51.22% 🍏
StringCache.java 50% -12.76% 🍏
XyzFeatureCollection.java 45.38% -2.64% 🍏
ConnectorPayload.java 42.91% 🍏
NakshaVersion.java 42.31% -3.85% 🍏
QueryParameterList.java 37.71% 🍏
StringHelper.java 35.65% -6.62% 🍏
Patcher.java 32.63% -4.49% 🍏
IoHelp.java 30.31% -8.68% 🍏
XyzProperties.java 29.9% -57.73% 🍏
ErrorResponse.java 25.34% -10.27% 🍏
XyzNamespace.java 25.16% 🍏
JsonMap.java 24.93% -4.04% 🍏
PluginCache.java 19.37% -80.16% 🍏
SimpleTask.java 17.33% -27.27% 🍏
Result.java 6.67% -57.78% 🍏
MapEntrySet.java 4.62% -11.92% 🍏
Changeset.java 0% 🍏
IoEventPipeline.java 0% -1.92% 🍏
EventPipeline.java 0% 🍏
NoCursor.java 0% 🍏
UncheckedException.java 0% -18.29% 🍏
Storage.java 0% -41.94% 🍏
XyzCollection.java 0% -8.51% 🍏
EventHandler.java 0% -53.03% 🍏
FibMapConflict.java 0% 🍏
ClosableRootResource.java 0% 🍏
CloseableResource.java 0% -39.42% 🍏
ClosableChildResource.java 0% 🍏
ResultHelper.java 0% -32.94% 🍏
RequestHelper.java 0% -33.77% 🍏
HereMetaNs.java 0% 🍏
XyzTags.java 0% 🍏
EReviewState.java 0% 🍏
XyzLabels.java 0% 🍏
HereDeltaNs.java 0% 🍏
EChangeState.java 0% 🍏
XyzCodec.java 0% 🍏
EExecutedOp.java 0% -45.61% 🍏
ForwardCursor.java 0% 🍏
SOpType.java 0% 🍏
SeekableCursor.java 0% 🍏
OpType.java 0% 🍏
MutableCursor.java 0% 🍏
SOp.java 0% 🍏
XyzCodecFactory.java 0% 🍏
POp.java 0% -76.11% 🍏
WriteRequest.java 0% -95.33% 🍏
XyzFeatureCodec.java 0% 🍏
WriteFeatures.java 0% -33.33% 🍏
XyzFeatureCodecFactory.java 0% 🍏
POpType.java 0% 🍏
Op.java 0% -31.08% 🍏
WriteXyzFeatures.java 0% -83.33% 🍏
FeatureBox.java 0% 🍏
XyzCollectionCodecFactory.java 0% 🍏
WriteCollections.java 0% -75% 🍏
FeatureCodec.java 0% 🍏
PRef.java 0% -85.07% 🍏
EWriteOp.java 0% -21.28% 🍏
WriteXyzCollections.java 0% 🍏
XyzCollectionCodec.java 0% -75% 🍏
ReadFeatures.java 0% -27.1% 🍏
CodecError.java 0% -88.89% 🍏
ModifyFeaturesReq.java 0% -26.83% 🍏
IStorage.java 0% -3.64% 🍏
IWriteSession.java 0% 🍏
ModifyFeaturesResp.java 0% -63.64% 🍏
CollectionInfo.java 0% -9.76% 🍏
DeleteOp.java 0% -75% 🍏
here-naksha-lib-psql EPsqlLogLevel.java 100% 🍏
PsqlFeatureGenerator.java 88.02% -11.98% 🍏
PsqlCursor.java 87.83% -6.96% 🍏
PostgresWriteFeaturesToPartition.java 67.57% 🍏
PsqlByUrlBuilder.java 67.39% -32.61% 🍏
PostgresCursor.java 44.12% 🍏
PsqlInstanceConfigBuilder.java 42.92% -57.08% 🍏
PsqlStorage.java 0% -93.64% 🍏
PsqlHelper.java 0% 🍏
NakshaOp.java 0% -44.19% 🍏
PsqlTxWriter.java 0% 🍏
EPsqlState.java 0% -94.07% 🍏
PostgresConnection.java 0% 🍏
PsqlReadSession.java 0% -31.25% 🍏
PsqlError.java 0% -39.29% 🍏
PsqlStorageSlf4jLogWriter.java 0% -85.71% 🍏
PostgresWriteOp.java 0% -88% 🍏
SQL.java 0% -16.32% 🍏
PostgresInstance.java 0% 🍏
PsqlInstance.java 0% 🍏
PsqlConnection.java 0% 🍏
PsqlStorageSlf4jWriter.java 0% 🍏
PostgresStorage.java 0% -88.41% 🍏
PsqlWriteSession.java 0% -27.69% 🍏
PsqlInstanceConfigDeserializer.java 0% 🍏
PsqlInstanceConfig.java 0% 🍏
PsqlShutdownTask.java 0% -13.04% 🍏
PsqlStorageProperties.java 0% 🍏
PsqlSession.java 0% -67.95% 🍏
PostgresSession.java 0% -85.78% 🍏
PsqlTxReader.java 0% 🍏
PsqlSuccess.java 0% -33.33% 🍏
DemoMain.java 0% 🍏
CleanInitMain.java 0% -32.43% 🍏

Copy link
Member

@hirenkp2000 hirenkp2000 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am approving PR for merge, once the review comments have been addressed (unless ofcourse, if there is difference in opinion, needing us to align separately).

.withDefaultSchema(NakshaAdminCollection.SCHEMA)
.build();
return new NakshaApp(config, cfgId, null);
final PsqlInstanceConfig config =
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

    .parseUrl(url)
    .withDefaultSchema(NakshaAdminCollection.SCHEMA)

Earlier code was ensuring that if schema is not provided as part of URL, then default schema naksha will be used. This allows us using different (custom) schema in the URL, even if using the same database instance.
Not sure if new change still supports this capability? But we surely need it.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am unaware of the reasons behind this change, we might readdress this after talking with Alex or Paweł about it

@@ -316,7 +429,7 @@ protected final void unlock() {

private static final AtomicLong threadCount = new AtomicLong();

private @NotNull RESULT init_and_execute() {
private @NotNull RESULT init_and_execute() throws Exception {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

avoid throwing exception, as this was the main reason we were seeing API requests hanging during pipeline execution.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed and fixed along with errorResult

RESULT result = null;
log.warn("The task failed with an exception. ", throwable);
return result;
protected @NotNull RESULT errorResponse(@NotNull Throwable throwable) throws Exception {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like the merge conflict is bringing back the old code. In the new change, function returns empty result and doesn't throw exception. This was added as a fix to ensure we don't get into API request hanging issue again we observed in pipeline.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

returned back to explicit null returning - though I think we need to tackle it with something else than null (maybe explicitly modelled error result?)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This class has significant changes compared to what was refactored in maintenance branch. Need to understand this further. Leave this open for now. I will check with Alex / Pawel about the changes and what can be done.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After having gone through all the changes, it looks like this has knock-on impact across files, if we have to make further changes in PluginCache. And it may not be worth blocking entire PR merge for the same. So, ignore my previous comment on this file and I am OK to go ahead with current changes.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As this class is applicable only for Naksha internal Admin resources (like Storage, EventHandler, Space and in future Subscription), perhaps worth renaming it to AdminFeatureEventHandler or something similar , to not confuse it with Custom Space based XyzFeatures.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice, renamed + added minor javadoc

}
return storage.newInstance(this);
final Storage storage =
Objects.requireNonNull(readFeatureFromResult(result, Storage.class), "Storage not found in result");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

eventhough minor thing, but let's retain the previous message (which prints the storageId)
("No storage found with id " + storageId)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Copy link

Code Coverage

Overall Project 31.41% -14.11% 🍏
Files changed 25.6% 🍏

Module Coverage
here-naksha-lib-heapcache 53.36% -3.89% 🍏
here-naksha-app-service 50.49% -0.77% 🍏
here-naksha-lib-hub 35.34% -14.68% 🍏
here-naksha-lib-core 30.04% -9% 🍏
here-naksha-lib-psql 20.73% -48.62% 🍏
Files
Module File Coverage
here-naksha-lib-heapcache HeapCache.java 79.06% 🍏
HeapFeatureWriter.java 50% -24.53% 🍏
here-naksha-app-service EventHandlerApiTask.java 93.87% 🍏
StorageApiTask.java 93.87% 🍏
SpaceApiTask.java 93.87% 🍏
WriteFeatureApiTask.java 82.65% 🍏
AccessLogUtil.java 81.47% 🍏
AbstractApiTask.java 73.57% 🍏
ReadFeatureApiTask.java 72.83% 🍏
NakshaApp.java 66.32% 🍏
NakshaHttpVerticle.java 55.28% -3.85% 🍏
StorageMaintainer.java 0% 🍏
here-naksha-lib-hub MockResult.java 100% 🍏
NHSpaceStorage.java 79.72% 🍏
NakshaHubFactory.java 75.61% 🍏
NHSpaceStorageWriter.java 67.65% -1.47% 🍏
NHAdminMock.java 58.2% -12.17% 🍏
NHSpaceStorageReader.java 53.33% 🍏
MockResultCursor.java 48.46% -24.62% 🍏
NHAdminWriterMock.java 33.9% -32.77% 🍏
NakshaHubMock.java 26.36% -26.36% 🍏
NHAdminReaderMock.java 3.25% -28.05% 🍏
NakshaHub.java 0% -15.68% 🍏
ConfigUtil.java 0% -15% 🍏
NHAdminStorage.java 0% 🍏
NHAdminStorageWriter.java 0% -30.95% 🍏
here-naksha-lib-core JsonEnumSerializer.java 100% 🍏
XyzError.java 93.06% -6.94% 🍏
Json.java 89.86% -1.09% 🍏
JsonModule.java 86.92% -3.74% 🍏
FibLinearProbeTable.java 86.52% 🍏
FibMapIterator.java 83.33% 🍏
ILike.java 82.93% -12.2% 🍏
JsonFieldByte.java 75.46% 🍏
FibRefType.java 74.19% 🍏
JsonFieldShort.java 73.77% 🍏
JsonFieldBool.java 72.11% 🍏
JsonConfigFile.java 71.34% -2.59% 🍏
JsonFieldDouble.java 71.03% 🍏
JsonFieldFloat.java 71.03% 🍏
JsonFieldInt.java 69.9% 🍏
JsonFieldLong.java 69.31% 🍏
JsonEnum.java 67.72% -7.61% 🍏
FibMap.java 64.81% -3.5% 🍏
JsonEnumDeserializer.java 63.04% 🍏
JsonFieldChar.java 62.03% -3.21% 🍏
FibSet.java 56.37% -2.49% 🍏
JsonSerializable.java 55.36% -2.77% 🍏
AbstractTask.java 54.89% -16.39% 🍏
JTSHelper.java 54.31% 🍏
JsonObject.java 51.22% 🍏
StringCache.java 50% -12.76% 🍏
XyzFeatureCollection.java 45.38% -2.64% 🍏
ConnectorPayload.java 42.91% 🍏
NakshaVersion.java 42.31% -3.85% 🍏
QueryParameterList.java 37.71% 🍏
StringHelper.java 35.65% -6.62% 🍏
Patcher.java 32.63% -4.49% 🍏
IoHelp.java 30.31% -8.68% 🍏
XyzProperties.java 29.9% -57.73% 🍏
ErrorResponse.java 25.34% -10.27% 🍏
XyzNamespace.java 25.16% 🍏
JsonMap.java 24.93% -4.04% 🍏
PluginCache.java 19.37% -80.16% 🍏
SimpleTask.java 17.33% -27.27% 🍏
Result.java 6.67% -57.78% 🍏
MapEntrySet.java 4.62% -11.92% 🍏
Changeset.java 0% 🍏
IoEventPipeline.java 0% -1.92% 🍏
EventPipeline.java 0% 🍏
NoCursor.java 0% 🍏
UncheckedException.java 0% -18.29% 🍏
Storage.java 0% -41.94% 🍏
XyzCollection.java 0% -8.51% 🍏
EventHandler.java 0% -53.03% 🍏
FibMapConflict.java 0% 🍏
ClosableRootResource.java 0% 🍏
CloseableResource.java 0% -39.42% 🍏
ClosableChildResource.java 0% 🍏
ResultHelper.java 0% -32.94% 🍏
RequestHelper.java 0% -33.77% 🍏
HereMetaNs.java 0% 🍏
XyzTags.java 0% 🍏
EReviewState.java 0% 🍏
XyzLabels.java 0% 🍏
HereDeltaNs.java 0% 🍏
EChangeState.java 0% 🍏
XyzCodec.java 0% 🍏
EExecutedOp.java 0% -45.61% 🍏
ForwardCursor.java 0% 🍏
SOpType.java 0% 🍏
SeekableCursor.java 0% 🍏
OpType.java 0% 🍏
MutableCursor.java 0% 🍏
SOp.java 0% 🍏
XyzCodecFactory.java 0% 🍏
POp.java 0% -76.11% 🍏
WriteRequest.java 0% -95.33% 🍏
XyzFeatureCodec.java 0% 🍏
WriteFeatures.java 0% -33.33% 🍏
XyzFeatureCodecFactory.java 0% 🍏
POpType.java 0% 🍏
Op.java 0% -31.08% 🍏
WriteXyzFeatures.java 0% -83.33% 🍏
FeatureBox.java 0% 🍏
XyzCollectionCodecFactory.java 0% 🍏
WriteCollections.java 0% -75% 🍏
FeatureCodec.java 0% 🍏
PRef.java 0% -85.07% 🍏
EWriteOp.java 0% -21.28% 🍏
WriteXyzCollections.java 0% 🍏
XyzCollectionCodec.java 0% -75% 🍏
ReadFeatures.java 0% -27.1% 🍏
CodecError.java 0% -88.89% 🍏
ModifyFeaturesReq.java 0% -26.83% 🍏
IStorage.java 0% -3.64% 🍏
IWriteSession.java 0% 🍏
ModifyFeaturesResp.java 0% -63.64% 🍏
CollectionInfo.java 0% -9.76% 🍏
DeleteOp.java 0% -75% 🍏
here-naksha-lib-psql EPsqlLogLevel.java 100% 🍏
PsqlFeatureGenerator.java 88.02% -11.98% 🍏
PsqlCursor.java 87.83% -6.96% 🍏
PostgresWriteFeaturesToPartition.java 67.57% 🍏
PsqlByUrlBuilder.java 67.39% -32.61% 🍏
PostgresCursor.java 44.12% 🍏
PsqlInstanceConfigBuilder.java 42.92% -57.08% 🍏
PsqlStorage.java 0% -93.64% 🍏
PsqlHelper.java 0% 🍏
NakshaOp.java 0% -44.19% 🍏
PsqlTxWriter.java 0% 🍏
EPsqlState.java 0% -94.07% 🍏
PostgresConnection.java 0% 🍏
PsqlReadSession.java 0% -31.25% 🍏
PsqlError.java 0% -39.29% 🍏
PsqlStorageSlf4jLogWriter.java 0% -85.71% 🍏
PostgresWriteOp.java 0% -88% 🍏
SQL.java 0% -16.32% 🍏
PostgresInstance.java 0% 🍏
PsqlInstance.java 0% 🍏
PsqlConnection.java 0% 🍏
PsqlStorageSlf4jWriter.java 0% 🍏
PostgresStorage.java 0% -88.41% 🍏
PsqlWriteSession.java 0% -27.69% 🍏
PsqlInstanceConfigDeserializer.java 0% 🍏
PsqlInstanceConfig.java 0% 🍏
PsqlShutdownTask.java 0% -13.04% 🍏
PsqlStorageProperties.java 0% 🍏
PsqlSession.java 0% -67.95% 🍏
PostgresSession.java 0% -85.78% 🍏
PsqlTxReader.java 0% 🍏
PsqlSuccess.java 0% -33.33% 🍏
DemoMain.java 0% 🍏
CleanInitMain.java 0% -32.43% 🍏

@Amaneusz Amaneusz merged commit f570e7a into Naksha_maintenance Nov 28, 2023
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants