Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add serverless provider #1374

Merged
merged 66 commits into from
Sep 7, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
66 commits
Select commit Hold shift + click to select a range
1dcbe0a
Draft for serverless
mrodm Jul 27, 2023
3f38c52
Ensure endpoints are healthy
mrodm Jul 27, 2023
4b3da63
Retrieve credentials
mrodm Aug 17, 2023
21ce687
Add status output
mrodm Aug 17, 2023
ef5e47e
Request credentials when creating project
mrodm Aug 17, 2023
c04ee5d
Move package to internal
mrodm Aug 17, 2023
f463215
Add elastic agent to serverless project
mrodm Aug 17, 2023
a54a1aa
Remove debug outputs
mrodm Aug 17, 2023
7f091eb
Create elasticsearch client in stack
mrodm Aug 21, 2023
98acfbd
Create kibana client in stack
mrodm Aug 21, 2023
3084d0a
Use kibana client from stack in serverless
mrodm Aug 21, 2023
27fff3c
Fix kibana client
mrodm Aug 21, 2023
57eb28e
Use kibana stack client to create agent and package policy
mrodm Aug 21, 2023
81b213c
Add method to get default fleet URL to kibana client
mrodm Aug 21, 2023
4e800e3
Move creation of agent policy to project
mrodm Aug 22, 2023
3302ca2
Fleet should be just checked if project is observability
mrodm Aug 22, 2023
54395dc
Use raw http request
mrodm Aug 22, 2023
bf42d9f
Use http client for Fleet
mrodm Aug 22, 2023
7850aab
Merge remote upstream/main into add_provider
mrodm Aug 22, 2023
4c448a8
Fix lint
mrodm Aug 22, 2023
8ff731a
Retrieve local services from docker daemon directly
mrodm Aug 23, 2023
30055e1
Update kibana.CheckHealth comment
mrodm Aug 23, 2023
d3df203
Update return in ResetCredentials
mrodm Aug 23, 2023
14fd7ce
Update returns
mrodm Aug 23, 2023
2060799
Keep just apiKey for serverless client struct
mrodm Aug 23, 2023
eea6fe3
Add DefaultFleetServerURL comment
mrodm Aug 23, 2023
cd3132a
Workaround to filter geo ip related keys
mrodm Aug 23, 2023
ea1328f
Remove unused import
mrodm Aug 23, 2023
7a7793e
Changes from code review
mrodm Aug 25, 2023
899495a
Changes from code review
mrodm Aug 25, 2023
bf7d6f3
Add allowed serverless project types
mrodm Aug 25, 2023
e211658
Remove leftovers
mrodm Aug 25, 2023
f68331e
Move clients to serverless provider
mrodm Aug 25, 2023
472095f
Reuse elastic agent env template
mrodm Aug 25, 2023
d09f0d7
Update stack templates with fleet_url and kibana_host
mrodm Aug 25, 2023
995a14a
Updated geo ip keys from ECS
mrodm Aug 25, 2023
5c62733
No need to return project in ResetCredentials
mrodm Aug 25, 2023
6416e0f
Merge remote-tracking branch 'upstream/main' into add_provider
mrodm Aug 30, 2023
1165923
Use registry.Production client
mrodm Aug 31, 2023
97b5a4f
Remove unused var
mrodm Aug 31, 2023
5bf8c32
Merge remote-tracking branch 'upstream/main' into add_provider
mrodm Sep 4, 2023
2644090
Update region
mrodm Sep 4, 2023
aa3f8d4
Add config examples in profile
mrodm Sep 4, 2023
efa4f76
Allow to loop for every event even with empty config
mrodm Sep 4, 2023
cc85e3c
Add two more GeoIP keys to be skipped
mrodm Sep 4, 2023
948f902
Return earlier if there is no need to loop for every event
mrodm Sep 4, 2023
9b669bf
Not return error if project does not exist - status command
mrodm Sep 5, 2023
44e3b48
Rename environment variables
mrodm Sep 5, 2023
ed9c77a
Add default host URL
mrodm Sep 5, 2023
e695477
Use serverless.ErrPrtojectNotExist error
mrodm Sep 5, 2023
ff9cd38
Set allowed project types as a list
mrodm Sep 5, 2023
57e890f
Add new profile config to set elastic cloud host URL
mrodm Sep 5, 2023
b54e095
Use default host url is env. var is empty
mrodm Sep 5, 2023
da1761a
Fix condition allowed project type
mrodm Sep 5, 2023
e95d1e5
Raise project not exist if any config parameters do not exist
mrodm Sep 5, 2023
8717537
Merge remote-tracking branch 'upstream/main' into add_provider
mrodm Sep 5, 2023
6bab96d
Use same compose file as in compose provider (snapshot.yml)
mrodm Sep 5, 2023
1ad954e
Retry docker-compose up once if elastic-agent failed
mrodm Sep 5, 2023
74b90d2
Add project type into status output
mrodm Sep 6, 2023
5d582b5
Add example config value
mrodm Sep 6, 2023
acfce7d
Fix typo in config.yml.example
mrodm Sep 6, 2023
a24aa47
Apply suggestions from code review
mrodm Sep 7, 2023
02648f8
Rename variables and make it private
mrodm Sep 7, 2023
582bfa9
Show elastic cloud endpoint always as debug
mrodm Sep 7, 2023
6757dd3
Ensure preference between config and environment variables for host URL
mrodm Sep 7, 2023
e90f283
Remove variables
mrodm Sep 7, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions internal/kibana/client.go
Original file line number Diff line number Diff line change
Expand Up @@ -52,11 +52,11 @@ func NewClient(opts ...ClientOption) (*Client, error) {
// Allow to initialize version from tests.
var zeroVersion VersionInfo
if c.semver == nil || c.versionInfo == zeroVersion {
v, err := c.requestVersion()
v, err := c.requestStatus()
if err != nil {
return nil, fmt.Errorf("failed to get Kibana version: %w", err)
}
c.versionInfo = v
c.versionInfo = v.Version

c.semver, err = semver.NewVersion(c.versionInfo.Number)
if err != nil {
Expand Down
45 changes: 45 additions & 0 deletions internal/kibana/fleet.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
// Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
// or more contributor license agreements. Licensed under the Elastic License;
// you may not use this file except in compliance with the Elastic License.

package kibana

import (
"encoding/json"
"errors"
"fmt"
"net/http"
)

// DefaultFleetServerURL returns the default Fleet server configured in Kibana
func (c *Client) DefaultFleetServerURL() (string, error) {
path := fmt.Sprintf("%s/fleet_server_hosts", FleetAPI)

statusCode, respBody, err := c.get(path)
if err != nil {
return "", fmt.Errorf("could not reach fleet server hosts endpoint: %w", err)
}

if statusCode != http.StatusOK {
return "", fmt.Errorf("could not get status data; API status code = %d; response body = %s", statusCode, respBody)
}

var hosts struct {
Items []struct {
IsDefault bool `json:"is_default"`
HostURLs []string `json:"host_urls"`
} `json:"items"`
}
err = json.Unmarshal(respBody, &hosts)
if err != nil {
return "", fmt.Errorf("failed to decode response: %w", err)
}

for _, server := range hosts.Items {
if server.IsDefault && len(server.HostURLs) > 0 {
return server.HostURLs[0], nil
}
}

return "", errors.New("could not find the fleet server URL for this project")
}
31 changes: 24 additions & 7 deletions internal/kibana/status.go
Original file line number Diff line number Diff line change
Expand Up @@ -30,29 +30,46 @@ func (v VersionInfo) IsSnapshot() bool {

type statusType struct {
Version VersionInfo `json:"version"`
Status struct {
Overall struct {
Level string `json:"level"`
} `json:"overall"`
} `json:"status"`
}

// Version method returns the version of Kibana (Elastic stack)
func (c *Client) Version() (VersionInfo, error) {
return c.versionInfo, nil
}

func (c *Client) requestVersion() (VersionInfo, error) {
var version VersionInfo
func (c *Client) requestStatus() (statusType, error) {
var status statusType
statusCode, respBody, err := c.get(StatusAPI)
if err != nil {
return version, fmt.Errorf("could not reach status endpoint: %w", err)
return status, fmt.Errorf("could not reach status endpoint: %w", err)
}

if statusCode != http.StatusOK {
return version, fmt.Errorf("could not get status data; API status code = %d; response body = %s", statusCode, respBody)
return status, fmt.Errorf("could not get status data; API status code = %d; response body = %s", statusCode, respBody)
}

var status statusType
err = json.Unmarshal(respBody, &status)
if err != nil {
return version, fmt.Errorf("unmarshalling response failed (body: \n%s): %w", respBody, err)
return status, fmt.Errorf("unmarshalling response failed (body: \n%s): %w", respBody, err)
}

return status, nil
}

// CheckHealth checks the Kibana health
func (c *Client) CheckHealth() error {
status, err := c.requestStatus()
if err != nil {
return fmt.Errorf("could not reach status endpoint: %w", err)
}

return status.Version, nil
if status.Status.Overall.Level != "available" {
return fmt.Errorf("kibana in unhealthy state: %s", status.Status.Overall.Level)
}
return nil
}
10 changes: 10 additions & 0 deletions internal/profile/_static/config.yml.example
Original file line number Diff line number Diff line change
@@ -1,2 +1,12 @@
# Directory containing GeoIP databases for stacks managed by elastic-agent.
# stack.geoip_dir: "/path/to/geoip_dir/"
## Elastic Cloud
# Host URL
# stack.elastic_cloud.host: https://cloud.elastic.co

## Serverless stack provider
# Project type
# stack.serverless.type: observability
# Region where the Serverless project is going to be created
# stack.serverless.region: aws-us-east-1
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It can maybe make sense to add also a setting for the host stack.elastic_cloud.host: "https://cloud.elastic.co"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a new config parameter for that


Loading