Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Azure] [PlatformLogs] Fix pipeline for edge cases #12735

Merged
merged 12 commits into from
Mar 5, 2025
Merged

[Azure] [PlatformLogs] Fix pipeline for edge cases #12735

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
45ccf0f
fix pipeline
lucian-ioan Feb 12, 2025
228d7db
add changelog entry
lucian-ioan Feb 18, 2025
1a9ac20
Merge branch 'main' into fix_azure_platform_logs
lucian-ioan Feb 18, 2025
5fa1164
bump manifest version
lucian-ioan Feb 18, 2025
b26012b
fix typo
lucian-ioan Feb 25, 2025
d467c11
convert eventduration to long
lucian-ioan Feb 25, 2025
1d35f0d
rerun tests
lucian-ioan Feb 25, 2025
909fc10
move logic to default pipeline
lucian-ioan Mar 5, 2025
2223bef
Merge remote-tracking branch 'origin' into fix_azure_platform_logs
lucian-ioan Mar 5, 2025
460ce77
bump version
lucian-ioan Mar 5, 2025
acb366e
fix typo
lucian-ioan Mar 5, 2025
7d7d46e
change type: bugfix
lucian-ioan Mar 5, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions packages/azure/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
- version: "1.22.2"
changes:
- description: Fix Platform Logs pipeline inconsistent casing for subscription_id field.
type: bugfix
link: https://github.com/elastic/integrations/pull/12735
- version: 1.22.1
changes:
- description: Fix the custom storage container description for the Azure Logs integration v2.
Expand Down
2 changes: 1 addition & 1 deletion ...data_stream/platformlogs/_dev/test/pipeline/test-platformlogs-edgecases.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -117,4 +117,4 @@
]
}
]
}
}
2 changes: 1 addition & 1 deletion ...a_stream/platformlogs/_dev/test/pipeline/test-platformlogs-identity-raw.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,4 +46,4 @@
]
}
]
}
}
2 changes: 1 addition & 1 deletion ...ta_stream/platformlogs/_dev/test/pipeline/test-platformlogs-invalid-raw.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,4 +141,4 @@
]
}
]
}
}
2 changes: 1 addition & 1 deletion ...zure/data_stream/platformlogs/_dev/test/pipeline/test-platformlogs-kube.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,4 +41,4 @@
]
}
]
}
}
2 changes: 1 addition & 1 deletion ...azure/data_stream/platformlogs/_dev/test/pipeline/test-platformlogs-raw.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,4 +87,4 @@
]
}
]
}
}
2 changes: 1 addition & 1 deletion ...ata_stream/platformlogs/_dev/test/pipeline/test-platformlogs-remote-raw.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,4 +45,4 @@
]
}
]
}
}
18 changes: 6 additions & 12 deletions ...es/azure/data_stream/platformlogs/elasticsearch/ingest_pipeline/azure-shared-pipeline.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,7 @@ processors:
- grok:
field: azure.resource_id
patterns:
- /SUBSCRIPTIONS/%{SUBID:azure.subscription_id}/RESOURCEGROUPS/%{GROUPID:azure.resource.group}/PROVIDERS/%{PROVIDERNAME:azure.resource.provider}/NAMESPACES/%{NAMESPACE:azure.resource.namespace}/AUTHORIZATIONRULES/%{RULE:azure.resource.authorization_rule}
- /subscriptions/%{SUBID:azure.subscription_id}/resourceGroups/%{GROUPID:azure.resource.group}/providers/%{PROVIDERNAME:azure.resource.provider}/namespaces/%{NAMESPACE:azure.resource.namespace}/authorizationRules/%{RULE:azure.resource.authorization_rule}
- /(?i)subscriptions/%{SUBID:azure.subscription_id}/RESOURCEGROUPS/%{GROUPID:azure.resource.group}/PROVIDERS/%{PROVIDERNAME:azure.resource.provider}/NAMESPACES/%{NAMESPACE:azure.resource.namespace}/AUTHORIZATIONRULES/%{RULE:azure.resource.authorization_rule}
pattern_definitions:
SUBID: (\{){0,1}[0-9a-fA-F]{8}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{12}(\}){0,1}
GROUPID: .+
Expand All @@ -20,8 +19,7 @@ processors:
field: azure.resource_id
if: 'ctx.azure?.subscription_id == null'
patterns:
- /SUBSCRIPTIONS/%{SUBID:azure.subscription_id}/RESOURCEGROUPS/%{GROUPID:azure.resource.group}/PROVIDERS/%{PROVIDERNAME:azure.resource.provider}/%{NAME:azure.resource.name}
- /subscriptions/%{SUBID:azure.subscription_id}/resourceGroups/%{GROUPID:azure.resource.group}/providers/%{PROVIDERNAME:azure.resource.provider}/%{NAME:azure.resource.name}
- /(?i)subscriptions/%{SUBID:azure.subscription_id}/resourceGroups/%{GROUPID:azure.resource.group}/providers/%{PROVIDERNAME:azure.resource.provider}/%{NAME:azure.resource.name}
pattern_definitions:
SUBID: (\{){0,1}[0-9a-fA-F]{8}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{12}(\}){0,1}
GROUPID: .+
Expand All @@ -32,8 +30,7 @@ processors:
field: azure.resource_id
if: 'ctx.azure?.subscription_id == null'
patterns:
- /SUBSCRIPTIONS/%{SUBID:azure.subscription_id}/RESOURCEGROUPS/%{GROUPID:azure.resource.group}/PROVIDERS/%{PROVIDERNAME:azure.resource.provider}/%{NAME:azure.resource.name}
- /subscriptions/%{SUBID:azure.subscription_id}/resourceGroups/%{GROUPID:azure.resource.group}/providers/%{PROVIDERNAME:azure.resource.provider}/%{NAME:azure.resource.name}
- /(?i)subscriptions/%{SUBID:azure.subscription_id}/resourceGroups/%{GROUPID:azure.resource.group}/providers/%{PROVIDERNAME:azure.resource.provider}/%{NAME:azure.resource.name}
pattern_definitions:
SUBID: (\{){0,1}[0-9a-fA-F]{8}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{12}(\}){0,1}
GROUPID: .+
Expand All @@ -53,8 +50,7 @@ processors:
field: azure.resource_id
if: 'ctx.azure?.subscription_id == null'
patterns:
- /SUBSCRIPTIONS/%{SUBID:azure.subscription_id}/PROVIDERS/%{PROVIDERNAME:azure.resource.provider}
- /subscriptions/%{SUBID:azure.subscription_id}/providers/%{PROVIDERNAME:azure.resource.provider}
- /(?i)subscriptions/%{SUBID:azure.subscription_id}/providers/%{PROVIDERNAME:azure.resource.provider}
pattern_definitions:
SUBID: (\{){0,1}[0-9a-fA-F]{8}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{12}(\}){0,1}
PROVIDERNAME: ([A-Za-z])\w+.([A-Za-z])\w+\/([A-Za-z][^\/])\w+
Expand All @@ -63,8 +59,7 @@ processors:
field: azure.resource_id
if: 'ctx.azure?.subscription_id == null'
patterns:
- /SUBSCRIPTIONS/%{SUBID:azure.subscription_id}/RESOURCEGROUPS/%{GROUPID:azure.resource.group}
- /subscriptions/%{SUBID:azure.subscription_id}/resourceGroups/%{GROUPID:azure.resource.group}
- /(?i)subscriptions/%{SUBID:azure.subscription_id}/resourceGroups/%{GROUPID:azure.resource.group}
pattern_definitions:
SUBID: (\{){0,1}[0-9a-fA-F]{8}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{12}(\}){0,1}
GROUPID: .+
Expand All @@ -73,8 +68,7 @@ processors:
field: azure.resource_id
if: 'ctx.azure?.subscription_id == null'
patterns:
- /SUBSCRIPTIONS/%{SUBID:azure.subscription_id}
- /subscriptions/%{SUBID:azure.subscription_id}
- /(?i)subscriptions/%{SUBID:azure.subscription_id}
pattern_definitions:
SUBID: (\{){0,1}[0-9a-fA-F]{8}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{12}(\}){0,1}
ignore_failure: true
Expand Down
10 changes: 9 additions & 1 deletion packages/azure/data_stream/platformlogs/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,12 @@ processors:
field: azure.platformlogs.resourceId
target_field: azure.resource_id
ignore_missing: true
if : ctx?.azure?.resource_id == null
- rename:
field: azure.platformlogs.ResourceId
target_field: azure.resource_id
ignore_missing: true
if : ctx?.azure?.resource_id == null
- rename:
field: azure.platformlogs.Region
target_field: cloud.region
Expand Down Expand Up @@ -148,8 +154,9 @@ processors:
- convert:
field: azure.platformlogs.durationMs
target_field: event.duration
type: integer
type: long
ignore_missing: true
ignore_failure: true
if: "ctx?.azure?.platformlogs?.durationMs != null && ctx?.azure?.platformlogs?.durationMs instanceof String"
- remove:
field: azure.platformlogs.durationMs
Expand Down Expand Up @@ -284,6 +291,7 @@ processors:
- set:
field: event.kind
value: event
ignore_failure: true
- pipeline:
name: '{{ IngestPipeline "azure-shared-pipeline" }}'
- pipeline:
Expand Down
2 changes: 1 addition & 1 deletion packages/azure/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
name: azure
title: Azure Logs
version: "1.22.1"
version: "1.22.2"
description: This Elastic integration collects logs from Azure
type: integration
icons:
Expand Down