Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: Verbose logs after retrying the projection a number of times #1245

Merged
merged 2 commits into from
Jan 28, 2025
Merged

chore: Verbose logs after retrying the projection a number of times #1245

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0c61ec6
chore: Verbose logs after retrying the projection a number of times
johanandren Nov 15, 2024
c6c2345
Merge branch 'main' into wip-grpc-projection-error-logs
johanandren Jan 28, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions akka-projection-core/src/main/scala/akka/projection/internal/ProjectionSettings.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,11 @@ package akka.projection.internal
import scala.concurrent.duration.FiniteDuration
import scala.concurrent.duration._
import scala.jdk.DurationConverters._

import akka.actor.typed.ActorSystem
import akka.annotation.InternalApi
import akka.projection.HandlerRecoveryStrategy
import akka.projection.Projection
import akka.stream.Attributes.LogLevels
import akka.stream.RestartSettings
import com.typesafe.config.Config

Expand Down Expand Up @@ -50,7 +50,12 @@ private[projection] object ProjectionSettings {
val maxRestarts = restartBackoffConfig.getInt("max-restarts")
if (maxRestarts >= 0) RestartSettings(minBackoff, maxBackoff, randomFactor)
else RestartSettings(minBackoff, maxBackoff, randomFactor).withMaxRestarts(maxRestarts, minBackoff)
}
}.withLogSettings(
RestartSettings.LogSettings.defaultSettings
.withLogLevel(LogLevels.Warning)
// Once we have retried many times, it could still be a transient failure but is
// more likely to be a permanent problem, so increase verbosity/include full stack trace
.withVerboseLogsAfter(5))
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if this should perhaps be from config and tuneable or if just having a hardcoded sensible default is enough. With the default settings this means that it starts printing a full stack trace after retrying and backing off 5 times ( ~1 minute and 20 seconds.)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could also just explicitly define it in gRPC replication, and recommend/show setting it for direct gRPC usage instead of defaulting here.


new ProjectionSettings(
restartSettings,
Expand Down
4 changes: 0 additions & 4 deletions ...ojection-grpc/src/main/scala/akka/projection/grpc/consumer/scaladsl/GrpcReadJournal.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -376,11 +376,7 @@ final class GrpcReadJournal private (
.invoke(streamIn)
.recover {
case ex: akka.grpc.GrpcServiceException if ex.status.getCode == Status.Code.UNAVAILABLE =>
// this means we couldn't connect, will be retried, relatively common, so make it less noisy,
// Users still want to be able to figure out non-transient errors, so log with full exception details at debug
val port = clientSettings.servicePortName.getOrElse(clientSettings.defaultPort.toString)
if (log.isDebugEnabled)
log.debug(s"Connection to ${clientSettings.serviceName}:$port for stream id $streamId failed or lost", ex)
throw new ConnectionException(clientSettings.serviceName, port, streamId)

case th: Throwable =>
Expand Down
3 changes: 0 additions & 3 deletions akka-projection-grpc/src/main/scala/akka/projection/grpc/internal/ConnectionException.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,9 @@ package akka.projection.grpc.internal

import akka.annotation.InternalApi

import scala.util.control.NoStackTrace

/**
* INTERNAL API
*/
@InternalApi
private[akka] final class ConnectionException(host: String, port: String, streamId: String)
extends RuntimeException(s"Connection to $host:$port for stream id $streamId failed or lost, will be retried")
with NoStackTrace
Loading