Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add logs agent integ test #433

Open
wants to merge 31 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
21aa051
update config files
varunch77 Nov 18, 2024
734f8e0
Add integ test for Logs agent
varunch77 Nov 18, 2024
5ed1a93
Have the agent sleep after starting it
varunch77 Nov 19, 2024
96522cd
update flush time to 90s
varunch77 Nov 19, 2024
b98c612
Update flush time to 120 seconds
varunch77 Nov 19, 2024
bdd40b9
Add delay to ensure tag deletion propagates
varunch77 Nov 19, 2024
43d2ca6
Change order of tests
varunch77 Nov 19, 2024
9f28f03
Change order of tests again
varunch77 Nov 19, 2024
e1bb9ce
Enable instance metadata tags
varunch77 Nov 19, 2024
fd59713
Clean up unused code
varunch77 Nov 19, 2024
6e00757
Differentiate sleep times
varunch77 Nov 19, 2024
bb229f7
Merge branch 'main' into add-logs-agent-integ-test
varunch77 Nov 19, 2024
84cc3d3
Address nits and make minor changes
varunch77 Nov 22, 2024
7a3b428
UGet region from EC2 metadata instead of hardcoding
varunch77 Nov 22, 2024
e37d84a
Revert EC2 metadata change
varunch77 Nov 22, 2024
79744f1
Revert EC2 metadata change
varunch77 Nov 22, 2024
7950725
Address misc comments
varunch77 Nov 22, 2024
37cb8a5
update code to search for log group directly
varunch77 Nov 22, 2024
006a057
Revert to search by prefix
varunch77 Nov 22, 2024
eb95406
address comments
varunch77 Dec 4, 2024
df7abce
remove unused packageS
varunch77 Dec 4, 2024
8dcc2d4
use clients from the awsservice package
varunch77 Dec 4, 2024
9f5722b
Merge remote-tracking branch 'origin/main' into add-logs-agent-integ-…
varunch77 Dec 4, 2024
6f1e4cf
Make ValidateEntity a common package
varunch77 Dec 4, 2024
5bd4a65
modify ValidateLogs to be a function in a common package
varunch77 Dec 4, 2024
d116feb
Merge remote-tracking branch 'origin/main' into add-logs-agent-integ-…
varunch77 Dec 4, 2024
6c272ee
Increase flush time
varunch77 Dec 5, 2024
58f0a88
run linter
varunch77 Dec 5, 2024
a555d2c
Merge branch 'main' into add-logs-agent-integ-test
varunch77 Jan 9, 2025
5138c20
Merge branch 'main' into add-logs-agent-integ-test
varunch77 Jan 17, 2025
dfb7272
Merge branch 'main' into add-logs-agent-integ-test
varunch77 Feb 10, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Address misc comments
varunch77 committed Nov 22, 2024
commit 795072522768ef075f3068f93a5c84642193acc3
33 changes: 22 additions & 11 deletions test/cloudwatchlogs/publish_logs_test.go
Original file line number Diff line number Diff line change
@@ -43,7 +43,7 @@ const (
standardLogGroupClass = "STANDARD"
infrequentAccessLogGroupClass = "INFREQUENT_ACCESS"
cwlPerfEndpoint = "https://logs.us-west-2.amazonaws.com"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This relies on the fact that we are running the test in us-west-2...

I know we only run our tests in us-west-2 but can we detect the region on the host and then populate the endpoint accordingly so that in the future we can support other regions?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we may already be constrained by the region being hardcoded elsewhere: https://github.com/aws/amazon-cloudwatch-agent-test/blob/main/util/awsservice/constant.go#L58C1-L60C78. We would need to make edits there but I'm a little wary of what the blast radius of that would look like for our testing

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need to hardcode the endpoint here. The code I wrote originally had to use a custom endpoint because it was a beta endpoint but this is the public us-west-2 endpoint which the client will automatically resolve

pdxRegionalCode = "us-west-2"
pdxRegionalCode = "us-west-2"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

region hardcoding is not needed since entity feature is GA


entityType = "@entity.KeyAttributes.Type"
entityName = "@entity.KeyAttributes.Name"
@@ -275,6 +275,17 @@ func TestWriteLogsWithEntityInfo(t *testing.T) {
if err != nil {
t.Fatalf("Error occurred creating log file for writing: %v", err)
}

// Defer file closing and removal with error handling
defer func() {
if err := f.Close(); err != nil {
t.Errorf("Error occurred closing log file: %v", err)
}
if err := os.Remove(logFilePath + "-" + id.String()); err != nil {
t.Errorf("Error occurred removing log file: %v", err)
}
}()

common.DeleteFile(common.AgentLogFile)
common.TouchFile(common.AgentLogFile)

@@ -286,11 +297,9 @@ func TestWriteLogsWithEntityInfo(t *testing.T) {
time.Sleep(sleepForExtendedFlush)
common.StopAgent()
end := time.Now()
begin := end.Add(-sleepForExtendedFlush * 4)

ValidateEntity(t, instanceId, instanceId, &end, testCase.expectedEntity)

f.Close()
os.Remove(logFilePath + "-" + id.String())
ValidateEntity(t, instanceId, instanceId, &begin, &end, testCase.expectedEntity)
})
}
}
@@ -489,20 +498,20 @@ func checkData(t *testing.T, start time.Time, lineCount int) {
assert.NoError(t, err)
}

func ValidateEntity(t *testing.T, logGroup, logStream string, end *time.Time, expectedEntity expectedEntity) {
func ValidateEntity(t *testing.T, logGroup, logStream string, begin, end *time.Time, expectedEntity expectedEntity) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nonblocking but recommended: This function has a lot of overlaps with this function: https://github.com/aws/amazon-cloudwatch-agent-test/blob/main/test/entity/entity_test.go#L241

Maybe good idea to move ValidateEntity to a common package and just use for any entity validation tests in the future.

log.Printf("Validating entity for log group: %s, stream: %s", logGroup, logStream)

logGroupInfo, err := getLogGroup()
logGroupInfo, err := getLogGroup(logGroup)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can just use IsLogGroupExists from our common package: https://github.com/aws/amazon-cloudwatch-agent-test/blob/main/util/awsservice/cloudwatchlogs.go#L137

The helper functions that I wrote was because we had to create a new client in us-east-1 for beta purpose but they are no longer needed

for _, lg := range logGroupInfo {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there no way to get a specific log group when you provide the name?

Seems expensive to get all log groups in an account. We have a ton in our test account..

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think there's a way to just search for one log group (why not is beyond me..) but I updated this so it passes in the log group as a prefix to search by (commit 006a057)

if *lg.LogGroupName == logGroup {
log.Println("Log group " + *lg.LogGroupName + " exists")
break
}
}
assert.NoError(t, err)
begin := end.Add(-sleepForExtendedFlush * 4)

log.Printf("Query start time is " + begin.String() + " and end time is " + end.String())
queryId, err := getLogQueryId(logGroup, &begin, end)
queryId, err := getLogQueryId(logGroup, begin, end)
assert.NoError(t, err)
log.Printf("queryId is " + *queryId)
result, err := getQueryResult(queryId)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

getLogQueryId and getQueryResult are not needed anymore since we can just use default client now. There's a helper function here: https://github.com/aws/amazon-cloudwatch-agent-test/blob/main/util/awsservice/cloudwatchlogs.go#L202

@@ -614,10 +623,12 @@ func getQueryResult(queryId *string) ([][]types.ResultField, error) {
}
}

func getLogGroup() ([]types.LogGroup, error) {
func getLogGroup(logGroupName string) ([]types.LogGroup, error) {
attempts := 0
var logGroups []types.LogGroup
params := &cloudwatchlogs.DescribeLogGroupsInput{}
params := &cloudwatchlogs.DescribeLogGroupsInput{
LogGroupNamePrefix: aws.String(logGroupName),
}
for {
output, err := cwlClient.DescribeLogGroups(context.Background(), params)