-
Notifications
You must be signed in to change notification settings - Fork 6
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: implement size-limited JSON requests for save metrics #406
feat: implement size-limited JSON requests for save metrics #406
Conversation
b5aa7f1
to
f83f65a
Compare
a3eb589
to
bf13e17
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think that intermediate types like *Str
are removable.
diff --git a/agent/src/repository/metric_repository.rs b/agent/src/repository/metric_repository.rs
index 6dbc120..6eb03d3 100644
--- a/agent/src/repository/metric_repository.rs
+++ b/agent/src/repository/metric_repository.rs
@@ -34,6 +34,7 @@ pub trait MetricStoreRepository {
}
#[derive(Debug, Serialize, Clone, PartialEq)]
+#[serde(rename_all = "snake_case")]
pub enum MetricType {
CpuUsage,
MemoryUsage,
diff --git a/agent/src/services/studio.rs b/agent/src/services/studio.rs
index 90c6876..a0696ca 100644
--- a/agent/src/services/studio.rs
+++ b/agent/src/services/studio.rs
@@ -413,18 +413,6 @@ impl MessageActivityRepository for Studio {
}
}
-#[derive(Serialize)]
-struct MetricStr {
- metric_type: String,
- value: f32,
-}
-
-#[derive(Serialize)]
-struct MetricsWithTimestampStr {
- timestamp: chrono::DateTime<chrono::Utc>,
- metrics: Vec<MetricStr>,
-}
-
impl MetricStoreRepository for Studio {
async fn save(&self, request: VecDeque<MetricsWithTimestamp>) -> anyhow::Result<()> {
let mut metrics = request;
@@ -435,25 +423,13 @@ impl MetricStoreRepository for Studio {
let mut current_size = 0;
while let Some(m) = metrics.pop_front() {
- let metrics_with_timestamp_str = MetricsWithTimestampStr {
- timestamp: m.timestamp,
- metrics: m
- .metrics
- .iter()
- .map(|metric| MetricStr {
- metric_type: metric.metric_type.to_string(),
- value: metric.value,
- })
- .collect::<Vec<MetricStr>>(),
- };
-
- let item_size = serde_json::to_string(&metrics_with_timestamp_str)?.len();
+ let item_size = serde_json::to_string(&m)?.len();
if current_size + item_size > JSON_BODY_MAX_SIZE {
metrics.push_front(m);
break;
}
current_size += item_size;
- metrics_str.push(metrics_with_timestamp_str);
+ metrics_str.push(m);
}
let model = VerifiableCredentials::new(my_did, json!(metrics_str), chrono::Utc::now());
agent/src/services/studio.rs
Outdated
.collect::<Vec<MetricStr>>(), | ||
}; | ||
|
||
let item_size = serde_json::to_string(&metrics_with_timestamp_str)?.len(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it is better to assert that item_size does not exceed JSON_BODY_MAX_SIZE like this.
if item_size > JSON_BODY_MAX_SIZE {
anyhow::bail!("Invalid item size");
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for your feedback.
item_size
is size of JSON encoded each metrics.
current_size
is total size of JSON encoded request body.
So I was implemented comparison between current_size and JSON_BODY_MAX_SIZE.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think that your implementation has the implicit assumption: item_size
does not exceed JSON_BODY_MAX_SIZE
.
So I thought we should explicitly state that like below.
if item_size > JSON_BODY_MAX_SIZE {
anyhow::bail!("Invalid item size");
}
if current_size + item_size > JSON_BODY_MAX_SIZE {
metrics.push_front(m);
break;
}
Note: If any element has item_size > JSON_BODY_MAX_SIZE
, it will loop infinitely.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for your feedback.
I have fixed the program in the following commit:
1952571
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I commented on #406 (comment) .
During NodeX Studio's maintenance mode, the NodeX agent caches metrics in a VecDeque structure.
Once NodeX Studio becomes available again, the agent resumes sending these cached metrics.
However, there's a potential issue where the agent's cache size may exceed the maximum allowable size for JSON requests to NodeX Studio.
To address this, this PR implements a solution that creates size-limited JSON request chunks.
This approach ensures that all metrics are successfully transmitted, regardless of the cache size, by breaking down large data sets into manageable portions.