feat：配置模板的导入 #2711

Ambition9186 · 2023-10-26T09:25:56Z

配置模板支持zip、gz、tar压缩包导入
批量新增和修改配置模板

AlkaidChan · 2023-10-28T02:48:17Z

bcs-services/bcs-bscp/pkg/criteria/constant/key.go

+
+const (
+	// MaxFileSize 最大文件
+	MaxFileSize = 5 * 1024 * 1024


这里是上传的文本类型最大文件大小，改为 MaxUploadTextFileSize 更加见名知意

AlkaidChan · 2023-10-28T02:51:54Z

bcs-services/bcs-bscp/pkg/criteria/constant/key.go

+	// MaxFileSize 最大文件
+	MaxFileSize = 5 * 1024 * 1024
+	// ContentLength 文件大小
+	ContentLength = 100 * 1024 * 1024


同理，这里也改成 MaxUploadContentLength

AlkaidChan · 2023-10-28T02:52:42Z

bcs-services/bcs-bscp/pkg/criteria/constant/key.go

+	// ContentLength 文件大小
+	ContentLength = 100 * 1024 * 1024
+	// FileMode 文件模式
+	FileMode = "unix"


这个常量已经定义过了：
string(table.Unix)

AlkaidChan · 2023-10-28T03:08:29Z

bcs-services/bcs-bscp/pkg/dal/dao/template_revision.go

+	m := dao.genQ.TemplateRevision
+	// 根据templateIDs获取TemplateRevision中最大ID
+	// 通过最大ID查询
+	var templateId []struct {


这里可以直接 var templateRevisionIDs []uint32 吧，我理解如果最后查询出来只有一列值，是可以 Scan 到一个数组的吧

AlkaidChan · 2023-10-28T03:59:50Z

bcs-services/bcs-bscp/cmd/api-server/service/config_import.go

+func (c *configImport) uploadFile(kt *kit.Kit, filePath, rootDir string) (*types.FileInfo, error) {
+	var (
+		content  bytes.Buffer
+		fileType = "text"


"text" 已被定义为常量
string(table.Text)

AlkaidChan · 2023-10-28T06:31:21Z

bcs-services/bcs-bscp/pkg/protocol/data-service/data_service.proto

+
+message BatchUpsertTemplatesReq {
+    message Item {
+        pbtemplate.Template   templates            = 1;


这里 templates 和 template_revisions 都不是 repeated 类型，命名不需要复数形式

AlkaidChan · 2023-10-28T06:36:47Z

bcs-services/bcs-bscp/cmd/data-service/service/template_revision.go

@@ -239,6 +241,21 @@ func getLatestTmplRevisions(tmplRevisions []*table.TemplateRevision) map[uint32]

 func generateRevisionName() string {


这里生成版本名称，到秒级就够了吧。这里索引应该是 templateRevisionName + templateID，一次导入操作，分别是对不同的 template 生成新的版本，基本上是不会重复的

AlkaidChan · 2023-10-28T06:38:58Z

bcs-services/bcs-bscp/cmd/data-service/service/template.go

+	return nil
+}
+
+func (s *Service) validationData(grpcKit *kit.Kit, updateId []uint32,


这里名称改为 validateBatchUpsertTamplates

AlkaidChan · 2023-10-28T06:44:02Z

bcs-services/bcs-bscp/cmd/data-service/service/template.go

+		})
+	}
+
+	err := s.doTemplateRevision(kt, tx, toCreateTr)


修改名称为： s.doCreateTemplateRevisions

AlkaidChan · 2023-10-28T06:49:19Z

bcs-services/bcs-bscp/pkg/protocol/config-server/config_service.proto

@@ -309,6 +315,12 @@ service Config {
            get : "/api/v1/config/biz/{biz_id}/template_spaces/{template_space_id}/templates/list_not_bound"
        };
    }
+    rpc GetTemplateDetailByUniqueKey(GetTemplateDetailByUniqueKeyReq) returns (GetTemplateDetailByUniqueKeyResp) {
+        option (google.api.http) = {
+            post : "/api/v1/config/biz/{biz_id}/template_spaces/{template_space_id}/templates/detail"


根据蓝鲸最新接口规范，按 restful 风格定义接口，这里用 get 方法

AlkaidChan · 2023-10-30T12:34:42Z

bcs-services/bcs-bscp/pkg/protocol/config-server/config_service.proto

+
+message ListTemplateByTupleResp{
+    message Item {
+        pbtemplate.Template   templates            = 1;


这里也是哈，不是 repeated 类型不用复数形式命名

AlkaidChan · 2023-10-30T12:34:58Z

bcs-services/bcs-bscp/pkg/protocol/config-server/config_service.proto

+message ListTemplateByTupleResp{
+    message Item {
+        pbtemplate.Template   templates            = 1;
+        pbtr.TemplateRevision template_revisions   = 2;


AlkaidChan · 2023-10-30T12:49:30Z

bcs-services/bcs-bscp/cmd/data-service/service/template_revision.go

@@ -239,6 +240,20 @@ func getLatestTmplRevisions(tmplRevisions []*table.TemplateRevision) map[uint32]

 func generateRevisionName() string {
 	// Format the current time as YYYYMMDDHHMMSS
+	time.Sleep(time.Second)


这里不用 sleep，不然上传一个压缩包有100个配置得 sleep 100 秒

AlkaidChan · 2023-10-30T12:55:19Z

bcs-services/bcs-bscp/pkg/protocol/data-service/data_service.proto

+}
+
+message ListConfigItemByTupleResp {
+    repeated pbci.ConfigItem config_item = 1;


这里命名需要用复数形式

AlkaidChan · 2023-10-30T13:00:02Z

bcs-services/bcs-bscp/cmd/api-server/service/config_import.go

+		fileType = string(table.Binary)
+	} else {
+		// 读取文件内容的前512个字节
+		buffer := make([]byte, 512)


这里只有5M以下就全部读进内存，只读 512 可能判断不准确，并且可能截断 utf8 的编码导致将文本文件误识别为二进制文件

之后还需要验证一下如果并发读大文件缓存过快增长会不会导致 OOM，如果会导致OOM，可以考虑限制并发处理文件的数量，以及用 sync.Pool 创建一个可复用的 buffer 池

AlkaidChan · 2023-11-07T02:31:51Z

bcs-services/bcs-bscp/cmd/api-server/service/config_import.go

+	"bscp.io/pkg/types"
+)
+
+var bufferPool sync.Pool


bufferPool 的 Get 方法一般是在全局变量定义的时候就声明了，不确定延迟声明会不会导致不同请求中使用的是不同的 bufferPool 对象，这个改为
var bufferPool = sync.Pool {
New: func() interface{} {
return new(bytes.Buffer)
},
}

* github-bk-bcs/master: upd: 访问服务管理列表页时清空本地上次访问服务记录 (#2732) fix：修复部分空间样式和交互问题 (#2730) 配置模板压缩包导入以及批量新增配置模板 (#2711) optimize: adjust bscp variable validation rule (#2729) fix: 加固集群重启kubelet的方式。 (#2728)

* kv 相关接口 * kv 相关接口 * 配置模板压缩包导入以及批量新增配置模板 (#2711) * kv 相关接口 * golint * golint * kv string 不允许换行 --------- Co-authored-by: ambition <[email protected]>

Ambition9186 force-pushed the bcs-bscp-config branch from 8b0e922 to 1719cdd Compare October 26, 2023 09:56

AlkaidChan added the bcs-bscp feature or bugfix for bscp label Oct 26, 2023

AlkaidChan reviewed Oct 28, 2023

View reviewed changes

Ambition9186 force-pushed the bcs-bscp-config branch 2 times, most recently from a29a977 to 2d2a8fb Compare October 30, 2023 09:33

AlkaidChan reviewed Oct 30, 2023

View reviewed changes

Ambition9186 force-pushed the bcs-bscp-config branch 4 times, most recently from d36d1b1 to 669c668 Compare November 6, 2023 07:13

AlkaidChan reviewed Nov 7, 2023

View reviewed changes

配置模板压缩包导入以及批量新增配置模板

e5cf554

Ambition9186 force-pushed the bcs-bscp-config branch from 669c668 to e5cf554 Compare November 7, 2023 06:14

AlkaidChan approved these changes Nov 7, 2023

View reviewed changes

AlkaidChan merged commit 5bff946 into TencentBlueKing:master Nov 7, 2023

Ambition9186 deleted the bcs-bscp-config branch January 5, 2024 02:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat：配置模板的导入 #2711

feat：配置模板的导入 #2711

Ambition9186 commented Oct 26, 2023

AlkaidChan Oct 28, 2023

AlkaidChan Oct 28, 2023

AlkaidChan Oct 28, 2023

AlkaidChan Oct 28, 2023

AlkaidChan Oct 28, 2023

AlkaidChan Oct 28, 2023

AlkaidChan Oct 28, 2023

AlkaidChan Oct 28, 2023

AlkaidChan Oct 28, 2023

AlkaidChan Oct 28, 2023

AlkaidChan Oct 30, 2023

AlkaidChan Oct 30, 2023

AlkaidChan Oct 30, 2023

AlkaidChan Oct 30, 2023

AlkaidChan Oct 30, 2023

AlkaidChan Oct 30, 2023

AlkaidChan Nov 7, 2023

		@@ -239,6 +241,21 @@ func getLatestTmplRevisions(tmplRevisions []*table.TemplateRevision) map[uint32]

		func generateRevisionName() string {

feat：配置模板的导入 #2711

feat：配置模板的导入 #2711

Conversation

Ambition9186 commented Oct 26, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment