Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor helper binaries to save 161MB of disk space when the agent is installed and reduce RPM by 48MB #1454

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Refactor helper binaries to save 161MB of disk space when the agent is installed and reduce RPM by 48MB #1454

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
7af0524
work in progress -- refactor config-translate in to amazon-cloudwatch…
chadpatel Dec 3, 2024
f5f3198
move the old unit tests to cmdutil tests
chadpatel Dec 3, 2024
aa25c3f
Move wizard and downloader to the amazon-cloudwatch-agent binary to f…
chadpatel Dec 16, 2024
5986b7f
fix bugs and linter errors
chadpatel Dec 17, 2024
6c4f2af
nosec G306 for 644 file
chadpatel Dec 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions cmd/amazon-cloudwatch-agent/amazon-cloudwatch-agent.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ import (
"syscall"
"time"

"github.com/aws/amazon-cloudwatch-agent/translator/cmdutil"
"github.com/influxdata/telegraf/agent"
"github.com/influxdata/telegraf/config"
"github.com/influxdata/telegraf/logger"
Expand Down Expand Up @@ -95,6 +96,16 @@ var fRunAsConsole = flag.Bool("console", false, "run as console application (win
var fSetEnv = flag.String("setenv", "", "set an env in the configuration file in the format of KEY=VALUE")
var fStartUpErrorFile = flag.String("startup-error-file", "", "file to touch if agent can't start")

// config-translator
var fConfigTranslator = flag.Bool("config-translator", false, "run in config-translator mode")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I spent some time trying to reduce the copy/paste aspect. Some of these constants are defined twice. There are some things to do to clean it up but they all end up less readable and add a lot of indirection. Open to ideas tho.

If I was forced to make one change I would put the descriptions in a constant

var fTranslatorOs = flag.String("ct-os", "", "Please provide the os preference, valid value: windows/linux.")
var fTranslatorInput = flag.String("ct-input", "", "Please provide the path of input agent json config file")
var fTranslatorInputDir = flag.String("ct-input-dir", "", "Please provide the path of input agent json config directory.")
var fTranslatorOutput = flag.String("ct-output", "", "Please provide the path of the output CWAgent config file")
var fTranslatorMode = flag.String("ct-mode", "ec2", "Please provide the mode, i.e. ec2, onPremise, onPrem, auto")
var fTranslatorConfig = flag.String("ct-config", "", "Please provide the common-config file")
var fTranslatorMultiConfig = flag.String("ct-multi-config", "remove", "valid values: default, append, remove")

var stop chan struct{}

func reloadLoop(
Expand Down Expand Up @@ -607,6 +618,16 @@ func main() {
}
}
return
case *fConfigTranslator:
ct, err := cmdutil.NewConfigTranslator(*fTranslatorOs, *fTranslatorInput, *fTranslatorInputDir, *fTranslatorOutput, *fTranslatorMode, *fTranslatorConfig, *fTranslatorMultiConfig)
if err != nil {
log.Fatalf("E! Failed to initialize config translator: %v", err)
}
err = ct.Translate()
if err != nil {
log.Fatalf("E! Failed to translate config: %v", err)
}
return
}

if runtime.GOOS == "windows" && windowsRunAsService() {
Expand Down
156 changes: 40 additions & 116 deletions cmd/config-translator/translator.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,137 +1,61 @@
// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
// SPDX-License-Identifier: MIT

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can bring this back, lost it when moving code around

package main

import (
"errors"
"flag"
"fmt"
"log"
"os"
"os/user"
"path/filepath"

"github.com/aws/amazon-cloudwatch-agent/cfg/commonconfig"
userutil "github.com/aws/amazon-cloudwatch-agent/internal/util/user"
"github.com/aws/amazon-cloudwatch-agent/translator"
"github.com/aws/amazon-cloudwatch-agent/translator/cmdutil"
"github.com/aws/amazon-cloudwatch-agent/translator/context"
"github.com/aws/amazon-cloudwatch-agent/translator/translate/otel/pipeline"
translatorUtil "github.com/aws/amazon-cloudwatch-agent/translator/util"
)
"os/exec"

const (
exitErrorMessage = "Configuration validation first phase failed. Agent version: %v. Verify the JSON input is only using features supported by this version.\n"
exitSuccessMessage = "Configuration validation first phase succeeded"
version = "1.0"
envConfigFileName = "env-config.json"
yamlConfigFileName = "amazon-cloudwatch-agent.yaml"
"github.com/aws/amazon-cloudwatch-agent/tool/paths"
)

func initFlags() {
var inputOs = flag.String("os", "", "Please provide the os preference, valid value: windows/linux.")
var inputJsonFile = flag.String("input", "", "Please provide the path of input agent json config file")
var inputJsonDir = flag.String("input-dir", "", "Please provide the path of input agent json config directory.")
var inputTomlFile = flag.String("output", "", "Please provide the path of the output CWAgent config file")
var inputMode = flag.String("mode", "ec2", "Please provide the mode, i.e. ec2, onPremise, onPrem, auto")
var inputConfig = flag.String("config", "", "Please provide the common-config file")
var multiConfig = flag.String("multi-config", "remove", "valid values: default, append, remove")
flag.Parse()

ctx := context.CurrentContext()
ctx.SetOs(*inputOs)
ctx.SetInputJsonFilePath(*inputJsonFile)
ctx.SetInputJsonDirPath(*inputJsonDir)
ctx.SetMultiConfig(*multiConfig)
ctx.SetOutputTomlFilePath(*inputTomlFile)

if *inputConfig != "" {
f, err := os.Open(*inputConfig)
if err != nil {
log.Fatalf("E! Failed to open common-config file %s with error: %v", *inputConfig, err)
}
defer f.Close()
conf, err := commonconfig.Parse(f)
if err != nil {
log.Fatalf("E! Failed to parse common-config file %s with error: %v", *inputConfig, err)
}
ctx.SetCredentials(conf.CredentialsMap())
ctx.SetProxy(conf.ProxyMap())
ctx.SetSSL(conf.SSLMap())
translatorUtil.LoadImdsRetries(conf.IMDS)
}
translatorUtil.SetProxyEnv(ctx.Proxy())
translatorUtil.SetSSLEnv(ctx.SSL())

mode := translatorUtil.DetectAgentMode(*inputMode)
ctx.SetMode(mode)
ctx.SetKubernetesMode(translatorUtil.DetectKubernetesMode(mode))
type flagDef struct {
name string
value *string
defaultVal string
description string
}

/**
* config-translator --input ${JSON} --input-dir ${JSON_DIR} --output ${TOML} --mode ${param_mode} --config ${COMMON_CONFIG}
* --multi-config [default|append|remove]
*
* multi-config:
* default: only process .tmp files
* append: process both existing files and .tmp files
* remove: only process existing files
*/
func main() {
initFlags()
defer func() {
if r := recover(); r != nil {
// Only emit error message if panic content is string(pre-checked)
// Not emitting the non-handled error message for now, we don't want to show non-user-friendly error message to customer
if val, ok := r.(string); ok {
log.Println(val)
}
//If the Input JSON config file is invalid, output all the error path and error messages.
for _, errMessage := range translator.ErrorMessages {
log.Println(errMessage)
}
log.Printf(exitErrorMessage, version)
os.Exit(1)
}
}()
ctx := context.CurrentContext()
log.Printf("Starting config-translator, this will map back to a call to amazon-cloudwatch-agent")

flags := []flagDef{
{"os", nil, "", "Please provide the os preference, valid value: windows/linux."},
{"input", nil, "", "Please provide the path of input agent json config file"},
{"input-dir", nil, "", "Please provide the path of input agent json config directory."},
{"output", nil, "", "Please provide the path of the output CWAgent config file"},
{"mode", nil, "ec2", "Please provide the mode, i.e. ec2, onPremise, onPrem, auto"},
{"config", nil, "", "Please provide the common-config file"},
{"multi-config", nil, "remove", "valid values: default, append, remove"},
}

mergedJsonConfigMap, err := cmdutil.GenerateMergedJsonConfigMap(ctx)
if err != nil {
log.Panicf("E! Failed to generate merged json config: %v", err)
for i := range flags {
flags[i].value = flag.String(flags[i].name, flags[i].defaultVal, flags[i].description)
}
flag.Parse()

if !ctx.RunInContainer() {
// run as user only applies to non container situation.
current, err := user.Current()
if err == nil && current.Name == "root" {
runAsUser, err := userutil.DetectRunAsUser(mergedJsonConfigMap)
if err != nil {
log.Panic("E! Failed to detectRunAsUser")
}
cmdutil.VerifyCredentials(ctx, runAsUser)
args := []string{"-config-translator"}
for _, f := range flags {
if *f.value != "" {
// prefix ct so we do not accidentally overlap with other agent flags
args = append(args, fmt.Sprintf("-ct-%s", f.name), *f.value)
}
}

tomlConfigPath := cmdutil.GetTomlConfigPath(ctx.OutputTomlFilePath())
tomlConfigDir := filepath.Dir(tomlConfigPath)
yamlConfigPath := filepath.Join(tomlConfigDir, yamlConfigFileName)
tomlConfig, err := cmdutil.TranslateJsonMapToTomlConfig(mergedJsonConfigMap)
log.Printf("Executing %s with arguments: %v", paths.AgentBinaryPath, args)

cmd := exec.Command(paths.AgentBinaryPath, args...)

cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr

err := cmd.Run()
if err != nil {
log.Panicf("E! Failed to generate TOML configuration validation content: %v", err)
}
yamlConfig, err := cmdutil.TranslateJsonMapToYamlConfig(mergedJsonConfigMap)
if err != nil && !errors.Is(err, pipeline.ErrNoPipelines) {
log.Panicf("E! Failed to generate YAML configuration validation content: %v", err)
}
if err = cmdutil.ConfigToTomlFile(tomlConfig, tomlConfigPath); err != nil {
log.Panicf("E! Failed to create the configuration TOML validation file: %v", err)
}
if err = cmdutil.ConfigToYamlFile(yamlConfig, yamlConfigPath); err != nil {
log.Panicf("E! Failed to create the configuration YAML validation file: %v", err)
if exitErr, ok := err.(*exec.ExitError); ok {
log.Panicf("E! Translation process exited with non-zero status: %d, err: %v", exitErr.ExitCode(), exitErr)
}
log.Panicf("E! Translation process failed. Error: %v", err)
os.Exit(1)
}
log.Println(exitSuccessMessage)
// Put env config into the same folder as the toml config
envConfigPath := filepath.Join(tomlConfigDir, envConfigFileName)
cmdutil.TranslateJsonMapToEnvConfigFile(mergedJsonConfigMap, envConfigPath)
}
Loading
Loading